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Integrated STEm or Integrated STEM? 


Integrated STEM learning experiences are recognized 
in K—12 education as a way to help students apply content 
area knowledge, make connections between discipline 
areas, and to advance student understanding and knowl- 
edge as a whole (e.g., Furner & Kumar, 2007; National 
Research Council, 2012). In a meta-analysis of research of 
integrated approaches among science, technology, engi- 
neering, and mathematics, Becker and Park (2011) found 
that integrated approaches have shown positive learning 
outcomes for students. Similarly, Stohlmann, Moore, and 
Roehrig, (2012) found that integrated lessons allow for a 
more authentic treatment of mathematics and science con- 
tent. As a result, schools have changed practices and legis- 
lation has been created to support integrated STEM 
learning opportunities for K-12 students (Johnson, 2012). 

Although research has identified potential benefits for 
students who participate in integrated STEM learning 
experiences, other research has shown that K-12 STEM 
curricula do not necessarily enhance mathematics and sci- 
ence learning (Tran & Nathan, 2010). In some cases, the 
integration of mathematics content in a meaningful way 
has been a challenge for teachers who are creating and 
implementing integrated STEM lessons. For example, 
Frykholm and Glasson (2005) found that mathematics 
was simply used as a tool for computation and data collec- 
tion. “This perception of ‘mathematics as tools’ may fall 
short of how some in the mathematics education commu- 
nity define mathematics and promote its study in the K-12 
experience” (Frykholm & Glasson, 2005, p. 139). Integra- 
tion of mathematics as a tool could be considered a type 
of context integration (Glancy et al., 2014; Moore et al., 
2014) that could help students make connections between 
the STEM disciplines. At the same time, this type of inte- 
gration falls short in helping students understand grade- 
level mathematics learning standards (c.f. Larson, 2017). 


Challenges With Integrating Mathematics 
When developing integrated STEM lessons, including 
content to be learned can be challenging. “Combining 
content from multiple disciplines in a meaningful way is 
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no easy task, and it is important to ensure that the essential 
aspects of each of the different content areas are not lost 
through the process of integration” (Glancy et al., 2014). 
Many of the topics used for integrated STEM lessons cre- 
ate challenges for the inclusion of mathematics content 
from the beginning. Example topics include energy con- 
servation, water quality, robotics, and recycling. There is 
definitely mathematics involved in each of these topics. 
The challenge for teachers and curriculum developers is 
identifying standards-based and grade-level appropriate 
mathematics content that aligns with the chosen topic. 
Related to this challenge is that the mathematics content 
connections typically identified are measurement and data 
analysis (Welty, Katehi, Pearson, & Feder, 2008) which 
represent a small portion of the standards-based mathe- 
matics curriculum. Opportunities for learning numeracy, 
proportional reasoning, linear and non-linear relation- 
ships, probability, and mathematical modeling are often 
not incorporated into the integrated STEM lessons. 

A second challenge in creating integrated STEM les- 
sons is the need for content and pedagogical knowledge in 
each of the discipline areas (Wang, Moore, Roehrig, & 
Park, 2011). For example, a teacher who specializes in 
mathematics is likely to have difficulty creating a unit that 
includes standards-based and grade-level appropriate tech- 
nology, engineering, and science learning goals. It will 
also be difficult for the mathematics specialist to under- 
stand and address the specialized pedagogical knowledge 
for all four of the discipline areas. 


Addressing the Mathematics Integration Challenge: 
An Example 

From 2013 to 2015, a team of STEM educators and I 
worked with a Midwestern high school to create and 
implement content-themed integrated STEM curricular 
units that would address academic standards in the STEM 
discipline areas (Vandrevala, Thomas, & Walker, 2015). 
The partnering high school was departmental-based; hav- 
ing separate science, mathematics, and technology/engi- 
neering departments. All of the participating teachers 
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specialized in a specific content area. The high school had 
little experience with integrated STEM curriculum. The 
administration and teachers committed to the work 
because they identified integrated STEM as a way to 
improve student interest and achievement. Teachers par- 
ticipated in approximately 80 hours of professional devel- 
opment each year. The professional development work 
included a summer institute, implementation of curricular 
units, classroom observations by the STEM educators, 
and follow-up meetings during the school year. 

In the first year, teachers formed multi-disciplinary 
teams to develop curricular units using the STEM Integra- 
tion Curriculum Assessment (Guzey & Moore, 2013) as a 
framework of features for an integrated STEM unit. In 
year one, a teacher team could have an engineering 
teacher, a biology teacher, a chemistry teacher, an algebra 
teacher, and a geometry teacher. Teachers created and 
implemented lessons on weather and flight, continental 
drift, engineering design, and solar systems. There were 
two challenges identified by the teachers in the evaluation 
of the first year (Walker & Bayley, 2017). First, all teach- 
ers, especially the mathematics teachers, noted that it was 
difficult to include standards-based and grade-level mathe- 
matics content in the integrated units. Second, teachers 
noted that it was difficult to teach content and use peda- 
gogical strategies from the disciplines in which they did 
not specialize. 

To address the first-year challenges, two changes were 
made to the professional development work. First, teacher 
teams were narrowed by discipline area. In year two, 
teams had no more than three discipline areas represented 
and were aligned by grade level. For example, most tenth- 
grade students enrolled in geometry and chemistry classes, 
so the geometry and chemistry teachers formed a group. 
Second, groups were asked to identify mathematics con- 
tent before considering content from other STEM disci- 
plines. For example, the tenth-grade geometry and 
chemistry teachers started with the topic of similar trian- 
gles. From the topic of similar triangles, the teachers 
developed a lesson on rocket propulsion that included 
opportunities for students to learn about the Ideal Gas 
Law and engineering design. 

In the evaluation following the second year, teachers 
noted that they had an easier time creating integrated 
STEM lessons and that the content was better aligned with 
grade-level academic standards (Walker & Bayley, 2017). 
Part of the ease and perceived alignment is likely due to the 
teachers having completed a prior year of experience with 
the STEM Integration Curriculum Assessment. Surveys and 
interviews provided evidence that the teachers felt that 
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starting with a mathematics topic was a key to the improve- 
ments in the second year. 


Suggestions 
Integrated STEM activities and units can provide oppor- 
tunities for students to meaningfully learn standards-based 
content and make connections between the STEM disci- 
plines. Teachers and curriculum developers face challenges 
with addressing the learning of content from multiple disci- 
pline areas and with appropriate pedagogical strategies. 
There is evidence that mathematics content is a particular 
challenge. In the example, teachers and STEM education 
experts had additional success starting with a mathematics 
topic to create integrated STEM units. The following are 
suggestions to help teachers and curriculum developers cre- 
ate curriculum that will help students learn standards-based 
and grade-level appropriate STEM content: 
¢ Try starting with a grade-level standards-based math- 
ematics topic for an integrated STEM unit. 
¢ Write out explicit learning objectives for each of the 
STEM discipline areas represented in the curricular unit. 
¢ Use an integrated STEM framework (e.g. Guzey & 
Moore, 2013; Johnson, Peters-Burton, and Moore, 2016; 
Vasquez, Sneider, and Comer, 2013) to guide curriculum 
development and align pedagogical strategies. 
¢ Work closely with teachers and experts in different 
STEM areas to learn about the content and pedagogical 
strategies related to the STEM unit you are developing 
and/or implementing. 
William S. Walker, III 
Purdue University 
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A pilot study was conducted on a multimodal educational tool, Peruvian Food Chain Jenga (PFCJ), with 5th-grade 
students (N = 54) at a public charter school. The goal was to compare the effectiveness of the multimodal tool to a more 
traditional presentation of the same materials (food chain) using an experimental/control design. Data collection included 
a pretest/posttest and a “What I Did/What I Learned” response sheet. Quantitative analysis of pretest/posttest results 
showed both groups improved from pretest to posttest; however, there was no statistically significant difference between 
posttest results of experimental and control groups. Qualitative analysis of student open-ended responses indicated a 
difference between students who used the PFCJ and students in the control. The most striking difference occurred in how 
the students perceived the connectedness of species and the awareness of human impact. Our findings suggest that using a 
model such as PFCJ as a means of teaching and connecting scientific content with practices related to ecosystems is an 


effective method of engaging students in intelligent discussions about these topics. 


A persistent challenge in science teacher education is 
assisting in-service and preservice teachers in creating an 
optimum learning environment within their classrooms. 
To meet this challenge, science teacher educators need to 
know the content required by state and national standards, 
topics teachers consider to be important, areas teachers 
find challenging to teach, and tools best suited to help 
K-12 students learn. In addition, science teacher 
educators need to be able to help teachers see ways to 
connect the big ideas of science by using appropriate 
pedagogical approaches. 

Ecology (with the disciplinary core idea of interdependent 
relationships in ecosystems) (Next Generation Science 
Standards [NGSS Lead States], 2013; Texas Education 
Agency [TEA], n.d.) is one content area that is introduced 
over the elementary grades and revisited in middle and high 
school. Therefore, teachers at all grade levels need to have 
an understanding of ecological complexity and be able to 
teach about many aspects of ecosystems. Historically, the 
concepts related to ecology have held a high priority. In a 
study asking teachers to rate the importance of content areas 
for their students, food webs were rated as one of 15 most 
important items in the biology curricula (Finley, Stewart, & 
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Yarroch, 1982). More recently, the National Science 
Education Standards (NSES) (NRC, 1996), NGSS (NGSS 
Lead States, 2013), and Texas Essential Knowledge and 
Skills (TEKS) (TEA, n.d.) include the study of ecosystems 
as essential topics of study. However, student achievement 
data reveal that as late as middle and high school, students 
do not have a good understanding of population and system 
dynamics (Grotzer & Basca, 2003). 

Despite the curricular importance of food webs and 
system dynamics, many teachers lack sufficient literacy in 
ecology (Lambert, 2006, Payne & Zimmerman, 2010). 
Partial understanding and low confidence in teaching 
ecosystems suggests that teachers find this content area 
difficult to teach, possibly resulting in lessons that are 
teacher-directed and require limited thinking or problem- 
solving by the students. As with most topics, ecosystems 
and the interdependence of organisms can be taught using 
various tools and formats. Providing in-service and 
preservice teachers with alternative tools for teaching 
science topics is an important component of teacher 
education. As such, we developed the Peruvian Food Chain 
Jenga (PFCJ) as a tool to assist teachers with teaching 
ecosystem dynamics. 


Peruvian Food Chain Jenga 


A possible way for teachers to help students understand 
the role of food chains in ecosystems is through the use of 
games. Games may be models or simulations of real 
complex systems (Gobert, 2005, Gobert et al., 2011), 
thereby helping students understand the interactive 
component of models. However, science teacher educators 
should not advocate games as a way to engage students 
unless the games have been shown to provide an equal or 
greater educational experience when compared with 
traditional instruction. Before the PFCJ can be fully 
introduced to teachers for implementation into the 
classroom, its effectiveness as an educational tool must first 
be understood through research. The aim of this study was 
to test the PFCJ as a multimodal educational tool that 
utilizes a game format to determine if it is a viable 
alternative for teaching the interdependence of organisms in 
a marine food chain and ecological content. 


Literature Review 

We present four bodies of research below. First, we 
provide a review of literature which addresses the ecology 
and food chain concepts related to our study. This literature 
explores the importance and difficulty of learning about the 
interdependence of organisms in an ecosystem. Next, we 
examine the literature focused on models because we are 
interested in a tool that modeled ecological interdependence 
utilizing a multimodal, game-like format. Third, we review 
literature on the modes through which meaning-making and 
communication occur, with an emphasis on modeling. 
Finally, we present the status of educational games because 
the tool we utilize the existing framework of the Jenga° 
game to manipulate a marine food chain. 
Food Webs 

The interdependent relationships in ecosystems are a 
disciplinary core idea outlined by the NGSS (NGSS Lead 
States, 2013) and the TEKS (TEA, n.d.) for students within 
several grade bands. For instance, students are expected to 
learn that food webs show relationships of animals that eat 
animals and animals that eat plants. This expectation seems 
simple at first, but ecosystems are complex networks that 
involve numerous variables, delicately balanced systems of 
causes and effects, and progressing structures which play 
out over time in ways that cannot be directly observed 
(Jacobson & Wilensky, 2006; Manz, 2012). Understanding 
how ecosystems work provides a foundation for learning 
about core scientific concepts, such as the theory of 
evolution and the theory of conservation of energy (Lehrer 
& Schauble, 2012). 

Although educators consider food chains and food webs 
important topics, these subjects often accompany student 
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misconceptions (Grotzer & Basca, 2003; Ozkan, Tekkaya, 
& Geban, 2004). Middle and high school students tend to 
think in terms of individuals and not populations, focus 
more on animals than on other organisms, and do not think 
about the community as a system (Grotzer & Basca, 2003). 
Furthermore, students show a limited understanding of 
ecosystem interactions, viewing the environment as a 
passive surrounding only functioning to provide resources 
(Lehrer & Schauble, 2012). Some students believe that 
disturbances in the food web depict unidirectional effects, 
shown by a deficiency in tracing multiple and extended 
ways in which the system could be disrupted (Hogan, 
2000). For instance, students believe that modifications in 
one population may not affect another because they are too 
far apart within the food chain (Ozkan et al., 2004). 

The marine and aquatic ecosystems have been neglected 
in K-12 science education despite their critical roles. 
Studies with a focus on education pertaining to aquatic and 
marine ecosystems have found a lack of knowledge and 
awareness of these ecosystems and their resources across 
the United States (Tran, Payne, & Whitley, 2010). 
Historically, the pictures in student textbooks have depicted 
food webs with arrows showing which organism was eaten 
as opposed to the direction of energy/matter flow 
(Schollum, 1983). More recent textbooks have been found 
to not always illustrate the complexity and interdependence 
of food chains and food webs (Beals, Krall, & Wymer, 
2012; Lin & Hu, 2003; Songer, Kelcey, & Gotwals, 2009). 

The study of food webs and food chains can support the 
practice and experience students need to look at the natural 
world as a complex, interrelated system. The inclusion of 
the ocean ecosystem in the curriculum has been found to 
promote student understanding of complex systems (Tran 
et al., 2010). Moreover, the study of food webs and chains 
offers solid opportunities for students to use, manipulate, 
and understand scientific models. Thus, teacher preparation 
programs should devote more attention to this topic. 
Teachers should acquire pedagogical and _ content 
knowledge that guarantees the support of student scientific 
thinking about marine food webs and food chains. They 
should be equipped to expose students to activities that 
promote personal interest and motivation to learn more 
about ocean ecosystems. 

Scientific Models 

Models have been used in meaning-making and the 
communication of scientific ideas for centuries. Models used 
for scientific teaching and representation include scale 
models, symbolic models, mathematical models, and 
theoretical models (Harrison & Treagust, 2000; Schwarz 
et al., 2009). Furthermore, models may be representations of 
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either static-external visualizations or dynamic systems 
(Biffi, Hartweg, Stewart, Patterson, Simanek, & Weinburgh, 
2016a). Static-external visualizations are classified as 
diagrams and models, while dynamic systems are referred to 
as simulations (Gobert, 2005). 

Dynamic scientific models serve as helpful tools that can 
be used by students to visualize scientific concepts, 
problematize or broaden understanding of a concept, 
develop a shared conceptual meaning of core ideas, predict 
scientific phenomena, and explore possible solutions (Manz, 
2012). Even though models are used for many types of 
educational purposes, students pay more attention to them as 
objects to be interpreted, but not for the generation and 
testing of ideas (Lehrer & Schauble, 2006). The NGSS 
(NGSS Lead States, 2013) and the TEKS (TEA, n.d.) 
discuss the importance of using tangible models as a method 
to develop students’ understandings of both the nature of the 
scientific enterprise and disciplinary core ideas. 

Baek, Schwarz, Chen, Hokayem, and Zhan (2011) 
suggests that involving students in scientific modeling can 
play a vital role in helping them cultivate and evaluate 
explanations of the natural world. In selecting models to 
help students learn about ecosystems, it is important to find 
dynamic, modifiable structures that can be used to represent 
changes in the numerous variable of complex systems (de 
Ruiter, Wolters, Moore, & Winemiller, 2005). Grumbine 
(2012) reported that using colored Tinker Toys° as a 
modifiable structure to represent food-web concepts was an 
enjoyable activity for students, and useful in identifying 
students’ conceptual understandings. The Jenga® tower is 
another such structure that can be utilized in the classroom 
to represent the complexities of environmental dynamics. 
De Ruiter et al. (2005) affirm that Jenga® towers may be 
beneficial over other models, such as the classic stone-arch 
metaphor, when representing food chains due to their ease 
of use and flexible structure that allows for modification by 
students. 

Multimodal Representations 

In recent years, science educators have researched many 
ways meaning-making and communication occur in science 
(Hand, McDermott, & Prain, 2016). Learning science can 
be .considered multimodal since it requires students to 
connect different types of representations (Yore & Hand, 
2010). Several views of multimodality have been offered. 
One representation is the hybrid language of science, which 
is composed of natural language, mathematical expressions, 
visual representations, and manual-technical operations 
(Lemke, 2004). Another representation describes the 
components as being “descriptive (verbal, graphic, tabular), 
experimental, mathematical, figurative (pictorial, analogous, 
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metaphoric), and kinesthetic representations” (Murcia, 
2010, p. 24). A third representation uses only two groups: 
descriptive (text, mathematical equations, logical phrases) 
and depictive (pictures, diagrams, graphics) (Gunel & 
Yesildag-Hasancebi, 2016). In each case, the components 
emphasize representation to facilitate the meaning-making 
process of scientific concepts. 

Students often find difficulty in understanding the entire 
span of a given concept if they only view pictorial or 
textual representation (Bennett, 2011; Carolan, Prain, & 
Waldrip, 2008). However, combining multiple 
representation enables students to interpret the concept in 
different perspectives (diSessa, 2004; Hand et al., 2016; 
Tytler, Peterson, & Prain, 2006). When textual modes are 
supported with visual aids or figurative applications, 
students connect the scientific processes with the 
representations more meaningfully (Carey & Levin, 2002; 
Tippett, 2008). Additionally, “data reveal conclusively that 
meaning is made in all modes separately, and at the same 
time, that meaning is an effect of all the modes acting 
jointly” (Kress, Jewitt, Ogborn, & Tsatsarelis, 2001, p. 1). 
Each mode has its own function, having strengths and 
weaknesses in intelligibility, precision, and associative 
meaning, all necessary to move students toward more 
complex understandings. 

There are several examples of multimodal applications in 
the literature. Biffi et al. (2016a, 2016b) describe the use of 
the Jenga® tower as a multimodal tool to represent the food 
chain of the Peruvian marine ecosystem. Additional 
examples include an interactive website to conduct virtual 
experiments about force and distance (Waldrip, Prain, & 
Carolan, 2006), a computer software to experience the 
spacing and moving of the particles (Waldrip & Prain, 
2013), and a multimedia tool demonstrating the relationship 
between the Sun, Earth, and Moon (Brown, Murcia, & 
Hackling, 2013). 

Games 

Games are another a way to provide educational 
instruction. They may be fictional or representational and 
may serve as a model of the actual system. When used in 
educational setting, games result in positive learning 
outcomes (Mayo, 2007). Evidence indicates that games 
teach not only facts, but also reasoning that can be applied 
to real life situations (Mayo, 2007). As models of complex 
systems, games help clarify difficult issues and provide a 
context to complex thinking (Bochennek, Wittekindt, 
Zimmermann, & Klingebiel, 2007). 

Research has also shown the personal changes that can 
occur among students as a result of games. In an early study 
involving gaming, four motivational components (attention, 
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relevance, confidence, and satisfaction) increase without 
affecting performance (Klein & Freitag, 1991). Later 
studies support the positive motivation in addition to 
advancing cognitive functions (Baines & Slutsky, 2009; 
Pinder, 2008). Barab and Dede (2007) found that games 
which involve collaborative work “may act as a catalyst for 
change in students’ self-efficacy” (p. 3). Increased self- 
efficacy, the extent to which a person feels he/she can 
succeed in a given situation, has been positively associated 
with achievement and continued pursuit of science courses. 
Another outcome of using games in educational settings is 
a change in agency. Students become producers, rather than 
consumers, as they make choices with consequences. This 
does not typically happen in traditional instruction (Gee, 
2003). Increased agency, rather than specific content 
knowledge, may be very important in changing students’ 
environmental behaviors. 


Theoretical Framework 

Our research draws on Vygotsky’s sociocultural 
constructivist theory (Vygotsky, 1978) and semiotic 
affordances (Bezemer & Kress, 2016; Lemke, 2000; 
Pickering, 2007) as the theoretical framework guiding our 
investigation into the effectiveness of the PFCJ. Vygotsky 
stresses the collaborative nature of learning, specifically the 
role of the “other.” Others who are “more knowledgeable” 
serve as conduits for the tools of the culture, including 
language. When learners reach a cognitive level at which 
they are no longer able to progress without help, the more 
knowledgeable “other” has the task of offering guidance. 
This guidance may come as a provocative question, an 
answer to a question, or an additional resource. Interaction, 
itself a social endeavor, helps the learner reach a higher level 
of understanding than would be possible without guidance. 
This theory supports having students work in small groups 
as they share ideas and act as peer tutors. Vygotsky’s theory 
of learning also includes the role of cultural signs as tools 
for learning. This theory on the use of cultural signs 
highlights the importance of learning to use the words, 
sentence structure, and physical tools of a discipline. 

Semiotics, the study of symbols in meaning-making and 
communication, is important in connection with multimodal 
research. Human beings use a wide range of semiotic 
resources, often simultaneously, to provide access to similar 
messages in different ways. Non-linguistic systems work 
with linguistic systems to afford the student collection of 
communication tools (Bezemer & Kress, 2016; Hammond 
& Gibbons, 2005). Multiple research groups examine 
the link between science activity (multimodal) and meaning- 
making (Tang, Tan, & Yeo, 2011; Waldrip et al., 2006). 
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Additional research highlights the benefits of using 
multimodal tools for teaching science (Gillen, Littleton, 
Twiner, Staarman, & Mercer, 2007). 


Methods 

The PFCJ, a multimodal educational tool, was developed 
as an alternative way to engage upper elementary students 
in a lesson on food chains. A detailed explanation of how 
the model was designed to be used in the classroom has 
been described by Biffi et al. (2016a). After developing the 
educational tool, we conducted a validity study of the PFCJ 
to determine its appropriateness for use with classroom 
students (Biffi, Hartweg, de la Fuente, Patterson, Stewart, 
Patterson, Simanek, & Weinburgh, 2016b). Feedback 
received from the students indicated that the design of the 
PFCJ met its desired objectives of (a) being “kid-friendly,” 
(b) demonstrating the value of scientific models, (c) 
providing content in an interesting format, and (d) using 
evidence to construct arguments for cause and effect. The 
classroom teacher also found the PFCJ to be an effective, 
easy-to-use teaching model for her students. In response to 
questions that arose during the validity study (Biffi et al., 
2016b), we continued our research to evaluate the PFCJ for 
its impact on student learning through a pilot study. 
Particularly, we were interested in using a mixed-methods 
approach (Creswell, 2003; Creswell & Plano Clark, 2011). 
A concurrent nested design was used to seek information 
beyond the main research question (#1). This method gave 
priority to one method (for us, the standard content test) 
while embedding a method of hearing student voices. 

The research team for the pilot study consisted of seven 
science education graduate students and a science education 
professor. All members of the team met regularly to plan 
the research, develop the assessments, collect data, and 
conduct analysis. 

Research Questions 

The research presented here focuses on the experience of 
two sets of students during a lesson on a marine ecosystem. 
The research purpose was to use a coarse-grain approach to 
determine if fine-grain research on the use of the tool is 
warranted. More specifically, we tested the idea that one 
multimodal experience with the PFC] is as beneficial as one 
traditional lesson on student learning. by asking the 
following questions: 

(1) What is the quantitative difference in: 

a. postinstruction achievement scores between the 
control and experimental groups of 5th-grade 
students? 

b. gain from pretest to posttest between the control 
and experimental groups of Sth-grade students? 
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(2) What, if any, are the qualitative differences found on 
the “What I Did/What I Learned” inventory between the 
control and experimental for 5th-grade students? 
Intervention 

We created two lessons having the same objectives and 
the same features, but using different instructional 
formats. Both control and experimental groups used the 
Introductory Video, Event Cards, and Guide Sheet as 
outlined in Biffi et al., (2016b). These components 
comprised a well-planned lesson, which presented the 
concepts of scientific models, food chains, and keystone 
species. 

The two lessons began to differ after completing the 
Guide Sheet. Students in the control group were given a 
handout with a chart that contained a predetermined 
number of individuals for each species of the food chain. 
The chart had space for students to record the new 
number of individuals after each “move.” Students then 
drew an Event Card, discussed what effect the situation 
would have on the food chain, and recorded the decision 
on the chart by adding or subtracting individuals of one 
or more of the species. When any population reached 
zero, students stopped and talked within the group about 
the results. 

For the experimental group, students participated in 
the same lesson with one modification. Instead of a 
chart, they built a wooden tower to model the food 
chain. This group used the same Event Cards, but rather 
than add or subtract individuals from the food chain, 
they removed or added the appropriate blocks labeled 
with stickers indicating the trophic level. When the 
tower fell due to becoming too fragile, the students 
discussed their results. The alternative lesson had all the 
components found in the more traditional lesson, but 
with the inclusion of the use of the multimodal tool that 
modeled a food chain. _ 

Participants 

For this pilot study, we selected a public charter school 
within the Dallas/Fort Worth Metroplex. The 413 students 
enrolled in the school were predominately Hispanic 
(71%) and economically disadvantaged (71%). Twenty- 
six percent of the students were classified as English 
language learners. All 71 students in the Sth grade were 
asked to participate in the study and were provided with 
informed consent forms. Data were collected from 54 
students with signed consents; however, due to the limited 
sample size, specific demographic information was not 
collected. Twenty-eight students were assigned to the 
control group, while 26 students were assigned to the 
experimental group. 


School Science and Mathematics 


Data Collection 

The research team prepared an 11-item multiple-choice 
assessment using released content questions from the 
state’s science exams. In addition, the research team 
developed items based around the multimodality and 
sociocultural constructivist nature of the intervention to 
cover the modeling, food chain and food-web concepts, 
and cause-effect relationships. A science educator and 
scientist reviewed the assessment to provide content 
validation, and two elementary teachers reviewed it for 
readability. 

During the third week of August (2015-2016 school 
year), the preintervention assessment was given to all the 
5th-grade student participants. Students in periods 1, 3, and 
5 were assigned to be the control group, and periods 2, 4, 
and 6 were assigned to be the experimental group. All 
lessons were taught by a trained, science education graduate 
student. Immediately after each lesson, the 5th-grade 
students were asked to complete a “What I did/What I 
learned” inventory. On this inventory, students were asked 
to write descriptions of the process they used and the 
concepts they learned as a result. Two weeks after the 
lesson, the 5th-grade students completed a delayed posttest 
composed of the same questions as in the pretest. Consistent 
with embedded experimental design methodology 
(Creswell & Plano Clark, 2011), this approach provided 
qualitative data during the intervention with quantitative 
data collected preintervention/postintervention. 

Data Analysis 

Quantitative data were collected because this is the 
method used by schools to assess the progress of students. 
The multiple-choice content assessment was graded, and 
students received | point for each correct answer. The four 
open-ended questions were scored on a 0-3 scale, using a 
rubric developed by the team (Webb, Herman, & Webb, 
2007). Scores on the pretest and posttest were entered into 
SPSS (v. 20). The assumption that there was no difference 
between the preintervention experimental and control group 
was tested using a ¢-test. We then used a f-test to compare 
the postexperimental test scores to the control group test 
scores and a paired ¢-test to compare each student’s 
preexperimental and postexperimental test scores. 

Qualitative data were collected in order to discover 
nuanced differences in the two groups. Responses from 
“What I Did/What I Learned” were coded using two 
characteristic methods of grounded theory, initial coding, 
and axial coding (Saldana, 2011). During the first cycle of 
coding, preliminary codes were assigned to students’ 
responses (focus on the text) by each research team member 
independently. This process produced a list of first level 
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Table 1 

Summary Statistics for Student Scores 

Group Mean N SD SE 

Control Pre 13.21 28 3:672 1.072 
Post 16.25 28 6.186 1.169 

Experimental Pre 12,19 26 4,382 0.859 
Post 14.19 26 4.364 0.856 


concepts. The team met, compared codes, and agreed on the 
codes to move into cycle two. In the second and subsequent 
cycles of analysis, the team coded together. During this 
process, the provisional codes (focus on the concepts) were 
grouped, sorted, and relabeled. Eventually, the salient codes 
were assigned by group consensus. Trustworthiness was 
established through credibility, confirmability, and 
transferability. The process of individual coding followed by 
group discussion and consensus established credibility and 
confirmability, whereas, the rich descriptions used in the 
results section provides transferability. 


Findings 

The PFCJ addresses an important content topic often 
found on student achievement assessments. Food chains 
and webs have been suggested as a way of introducing 
Students to the system concept, by reflecting on energy 
flows or population changes (Hogan, 2000). Additionally, 
researchers have pointed out that it is important to analyze 
the complex nature of food webs when teaching ecosystem 
principles (Karamanos et al., 2012). Thus, both systematic 
thinking and ecosystem content can be taught as part of the 
same lesson on food chains. Having used a mixed-methods 
design, our findings will first present the quantitative data 
followed by the qualitative data, with discussion of the 
implications immediately following each question. 
Quantitative Data 

In response to research question la, “What is the 
quantitative difference in postinstruction achievement 
scores between the control and experimental groups of 
Sth-grade students?” the comparison of the posttest scores 
of both groups brought the following results. There was no 
statistically significant difference between the scores of the 
posttest of the control (16.25+6.186 points) and 


Table 2 

Paired-Samples Statistics for the Control and Experimental Group 

Group Average of Mean Differences N 
Control 3.04 28 
Experimental 2.00 26 


experimental group (14.19 + 4.364 points), 452) = 1.402, 
p = 0.167 (Table 1). This indicates that both experimental 
and control groups ended the unit with approximately the 
same content when assessed by our standard content test. 

In response to research question 1b, “What is the 
quantitative difference in gain from pretest to posttest 
scores between the control and experimental groups of 
5th-grade students?” the comparison of the scores of both 
groups brought the following results. In a comparison of 
paired-samples t-test between the pretest and posttest of 
each group, we found a highly significant difference 
between the pretest/posttest scores in both the control 
((27) = —3.457, p= 0.002) and the experimental group 
(t(25) = —2.733, p = 0.011) (Table 2). In the experimental 
group, there was a mean increase of 2.00 points in correctly 
answered questions between the experimental group’s 
pretest and posttest scores. Thus, the two lessons produced 
an increase in content knowledge about scientific models, 
food chains, and keystone species. 

Discussion. These results demonstrate that Sth-grade 
students do equally well on the content-based assessment 
when a _ well-planned traditional approach and _ the 
multimodal game-format approach are used. This is 
understandable as both sets of students utilized the same 
Event Cards (presenting content) and discussed the ideas in 
small groups. The difference was in manipulating the 
dynamic model, which utilized blocks and allowed students 
to see the tower become fragile and eventually collapse as 
populations were reduced. The results from the study 
indicate that the alternative lesson (PFCJ) did not interfere 
with student achievement and is, therefore, appropriate to 
use as an alternative to a traditional lesson. 

The PFCJ also addresses the issue of engaging students 
in experiences where they use and construct models as a 
way of presenting ideas and explanations which are 
important components of the most current science 
teaching standards (NGSS Lead States, 2013; TEA, n.d.). 
Because food chains and the connectedness of species 
within an ecosystem are complex in nature, utilizing 
dynamic models may be an alternative way of teaching 
the topic. PFCJ uses food webs as a model and the Event 
Cards with content as a springboard for constructing 
ideas about ecosystems. 


SD t-Value df Two-Tailed sig. 
5.672 =o) 2h 0.002 
S731 TSS Zo 0.011 
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Qualitative Data 

In response to research question 2, “What, if any, are the 
qualitative differences found on the “What I did/What I 
learned” inventory between the control and experimental 
groups of Sth-grade students?” student responses to the 
inventory were qualitatively coded between both groups, 
which brought the following results. Two major themes 
emerged: awareness and connectedness. Both themes were 
evident in the control and experimental groups, however, 
there were differences in the frequency themes and 
subthemes between the two groups. 

Awareness. The awareness theme had two subthemes, 
which were human impact and endangered species. Both 
groups of students gave examples of having gained an 
awareness of human impact. A few students in the control 
group also included the idea of having a new awareness 
about endangered species. For example, a student in the 
control group wrote “I learned what the populations could 
turn out to be like in the [future]. I learned that the animals 
are in danger and could decrease at any time in the world 
and be extinct.” 

Connectedness. The students who wrote about 
connectedness either demonstrated an understanding of the 
complex, holistic nature of the food chain, or the less 
complex predator—prey relationship. Although both groups 
mentioned connectedness, this theme was much more 
prevalent in the experimental group than in the control 
group. Students who were engaged in building the tower 
and watched it become unstable and eventually collapse 
due to different events, used language that indicates a more 
sophisticated understanding of the connectedness and 
interdependence of the species at twice the rate of control. 
As an example of connectedness, one student from the 
experimental group wrote: “I learned about the food chain. 
Also that all species depend on others to eat. A food chain 
starts when a small animal or plants gets eaten.” The 
control group did not appear to grasp the holistic connected 
nature of the food chain at the same level that was evident 
in the experimental group. To illustrate these ideas, 
examples of the themes are provided in Table 3. 

Discussion. The results from the elementary students’ 
written responses are especially interesting to educators 
when focusing on environmental issues. Both the 
experimental and control groups indicated that they now 
have a new awareness of human impact and endangered 
species. Issues like microplastics or ghost fishing, 
introduced to students in the Event Cards, promoted 
responses like “I learned that a plastic kills animals.” 
Having an instructional tool that has been shown to leave 
students aware of the human impacts to the ecosystem is 
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Table 3 

Themes Discovered in “What I Did/What I Learned” Inventory 

Theme Examples 

Awareness ; 
Subtheme: “T learned that the more species we kill 


the food chain will run out of food.” 
“T learned that a plastic kills animals.” 
“T learned what the populations could 
turn out to be like in the featur /sic/. I 
learned that the animals are indanger 
[sic] and could decrease at any time 
in the world and be extinct. I learned 
that adding and substracting /sic/ will 
be in my life even when I’m doing 
science. I learned there are animals 
that need help all over the world 
because they are getting murderd 
[sic] and they decreases the 
population in the animals life.” 


Human impact 


Subtheme: 
Endangered species 


Connectedness 
Subtheme: 
Predator—prey 


“T realized the more the top pradators 
[sic] decreased the bottom predadors 
[sic] increased” 

“Animals depend on each other. If one 
dies it affects everything.” 

“When one animal or species extinct, the 
whole life cycle will distrupt /sic/.” 

“T learned that it is important to take 
care of things because animals need 
to eat and they eat another and pass it 
on and if you don’t have what they 
eat they will die and also the other 
animals and pass it on.” 


Subtheme: Holistic 


important. The U.S. Environmental Protection Agency 
(EPA) recognizes public awareness as a main component of 
environmental education, which results in enhancing 
individuals’ “skills to make informed decisions and take 
responsible action” (EPA, 2016, para. 1). We believe the 
Event Cards, which presented both groups with content, 
attributed to this awareness. 

The most enlightening outcome was that the 
experimental group was qualitatively different from the 
control group in regard to the level of connectedness they 
expressed in their writings. The two subthemes of 
understanding of the complex holistic nature of a food 
chain and the less complex predator-prey relationship were 
stated by the experimental group. We attribute this 
additional and important understanding of system 
complexities to having witnessed the fragility of the tower. 
Unlike the control group, these students witnessed their 
model becoming unstable and falling apart as event cards 
provided situations that stressed the trophic levels. The 
PFCJ model helped solidify the understanding of 
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ecosystem connectedness in a way that the traditional 
instruction did not. These findings support the research 
from de Ruiter et al. (2005) that suggest the modifiable 
structure of the Jenga® tower can serve as an effective tool 
to represent changes in the numerous variables present in 
complex systems. Since prior research has shown that 
students have trouble understanding the interactions within 
ecosystems (Lehrer & Schauble, 2012), we think our 
findings indicate that the PFCJ may be an effective 
educational tool to use in the classroom when teaching 
ecosystems. 


Conclusions 

Our findings allow us to conclude that using a model such 
as PFCJ as a means of teaching and connecting scientific 
content with practices related to ecosystems is an effective 
method of engaging students in intelligent discussions about 
these topics. By promoting group debate and discourse 
about events and matters of scientific concern, PFCJ offered 
richness and excitement of learning about the natural world 
as well as rigorous content on critical environmental issues, 
and aided students in understanding the interconnectedness 
present in ecosystems. 

As teacher educators, we know that in many countries a 
major emphasis is placed on how to improve student 
achievement. However, in addition to student achievement, 
cognitive processes (i.e., critical thinking, synthesis, and 
analysis) and working in teams are being fore fronted as 
21st century skills. Therefore, in teacher preparation and 
professional development, we advocate teaching strategies 
that include using real-world events that have the potential 
to increase achievement, critical thinking, attitudes, and 
other student outcomes. However, before new strategies are 
provided to teachers, it is important to verify their 
effectiveness in the classroom. This pilot study serves as a 
critical step towards the development of the PFCJ as a tool 
to be used in future teacher development programs. 

As educational researchers, we understand how 
important it is to connect research with practice in the field 
of education. Consequently, our study contributes to the 
field of science teacher education by presenting a Jenga® 
multimodal tool that aligns with the S5th-grade TEKS (5.3 
critical thinking and scientific problem-solving; 5.9. 
Organisms and environments) and NGSS for disciplinary 
core concepts (LS2: Ecosystems) and crosscutting concepts 
(systems) as a way to model a marine ecosystem for upper 
elementary students. 

We consider our model a viable alternative way for 
teaching the interdependence of organisms in_ their 
environment. The alternative lesson we offer takes equal 
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time as the traditional lesson and is not expensive. The 
materials necessary to prepare this lesson have been 
outlined in Science Scope (Biffi et al., 2016a) and can be 
easily obtained by science teacher educators. 


Limitations and Areas of Future Research 

This pilot study contains limitations that may be 
addressed in future studies. First, this study was conducted 
with a limited sample size from a single grade level and 
school. Due to the limited size, specific demographic data 
of students was not collected during the study. Increased 
sample size and demographic data may provide additional 
insight in regards to the effectiveness of the PFCJ as an 
educational tool for subsets of students. Another major 
limitation is the duration of the intervention. The 
intervention occurred over a single lesson. While one 
lesson is often all that a teacher is able to give to the topic of 
food chains, future research may focus on the effectiveness 
of the PFCJ after multiple uses. Additionally, data 
collection was limited to student achievement tests and the 
analysis of “What I Did/What I Learned” inventory after 
the completion of the lesson. Discourse analysis of student 
conversations during the lesson could reveal nuanced 
differences that the written responses lost. With the positive 
results of the presented pilot study, the research team has 
designed future studies to examine the effectiveness of this 
tool in different settings. 
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This study examines Middle School Mathematics Teachers’ (MSMTs’) (N = 1,241) perceptions of the Common Core 
State Standards for Mathematics (CCSSM) and its impact on the instructional environment. A total of eight factors appear 
in the data. These factors include professional support, teachers’ use of district-adopted and non-district-adopted 
curricular resources, influence of CCSSM assessment and teacher evaluation on the instructional environment, influence 
of the CCSSM on classroom instruction, teachers’ planning practices, and perceived rigor of the CCSSM. The data 
suggest that teachers’ use of digital resources located online was disconnected from their district-adopted resources 
suggesting a lack of curricular coherence. MSMTs note that the CCSSM had caused them to incorporate more reform- 
oriented practices such as exploration and productive struggle into their daily instruction. MSMTs also perceive that the 
CCSSM includes new content that is more rigorous than previous state standards. Sampled MSMTs state that while state 
assessments will measure CCSSM content, they are less likely to include more complex problems or the standards for 
mathematical practice. Teachers are more likely to read teacher resources than student textbook activities online and to 
use digital resources for remediation instead of inquiry activities. Over one-third of MSMTs wanted more CCSSM 


professional development. 


For the first time in the history of the school mathematics 
education system in the United States, the majority of 
school students are under the umbrella of influence of one 
set of standards, the Common Core State Standards for 
Mathematics (CCSSM). The effects of the CCSSM are 
further magnified due to required assessments and the 
subsequent connection of these assessments to teacher 
evaluation (Duncan, 2009a, 2009b). Middle school 
mathematics teachers (MSMTs) are especially under 
increased pressure, as they are required to test students 
annually in Grades 6 through 8 on assessments aligned with 
the CCSSM. Furthermore, research conducted by Reys, 
Dougherty, Olson, and Thomas (2012) suggests that 
there are important differences between CCSSM middle 
school content standards and previous state curriculum 
frameworks. While a number of studies have been 
conducted on teachers’ perceptions of the CCSSM (e.g., 
Hart Research Associates, 2013) these results are not often 
disaggregated at the middle school level. Additionally, 
previous research has only superficially examined teachers’ 
perceptions of the Standards for Mathematical Practice 
(SMPs) (Perry et al., 2015) or the connections between the 
CCSSM and teachers’ curricular resources (Scholastic, 


2013). For these reasons, this study reveals the underlying 
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structure of a large sample of MSMTs’ (V= 1,241) 
perceptions of the CCSSM and its impact on the 
instructional environment. 


Background 

Changes in the CCSSM 

In their review of the influence of teaching on students’ 
learning, Hiebert and Grouws (2007) note that research 
suggests that opportunity to learn is an important necessary 
condition for student learning. Moreover, the focus of 
school mathematics instruction has often been broken down 
into two categories: teaching for skill efficiency and 
teaching for conceptual understanding. Recitation has been 
the predominant form of teaching in the United States and 
in school mathematics classroom this instruction has 
focused on skill efficiency (Stigler & Hiebert, 1999). 
Hiebert and Grouws note two features of teaching that 
promote conceptual understanding: teachers and students 
attend explicitly to concepts and students struggle with 
important mathematics. These features are connected to the 
CCSSM. For instance, McCallum, one of the architects of 
the CCSSM, writes that conceptual understanding is one of 
the components of rigor in the CCSSM (2012) along with 
procedural fluency and meaningful applications. The 
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importance of students persevering through struggles in 
solving problems appears as a habit of mathematically 
proficient students in the SMPs (Common Core State 
Standards Initiative, 2010; McCallum, 2012). 

Research suggests that the CCSSM is different from 
previous state mathematics standards (Porter, McMaken, 
Hwang, & Yang, 2011; Reys et al., 2012). Specifically, 
Reys et al. (2012) found that content appearing at the high 
school level in previous state standards now appeared at the 
middle school level while Porter et al. found low alignment 
between the CCSSM and previous state standards in terms 
of mathematics content as well as cognitive demand. 
Schmidt and Houang (2012) found the CCSSM to be very 
similar to the curriculum standards used by countries that 
were considered to be high achieving on the Third 
International Mathematics and Science Study administered 
in 1995. They also found that state standards that were used 
before the advent of the CCSSM were found to be less 
similar to standards from these high-achieving countries. 
Additionally, state standards that were more similar to the 
CCSSM performed better on the 2009 Grade 8 National 
Assessment of Educational Progress. Cobb and Jackson 
(2011) found the CCSSM to be focused in that different 
components of important mathematical ideas were 
addressed over time. In terms of quality, they found the 
CCSSM to have what Bruner (1987) termed “clout” due to 
their emphasis on applying mathematics to contexts outside 
of school mathematics within the content standards as well 
as the modeling SMP. Cobb and Jackson also used 
coherency to evaluate the quality of the CCSSM and agree 
with the work by Confrey, Maloney, and Nguyen (2010) 
showing how leaming progressions build students’ 
understanding of central mathematical ideas across grades. 
Cobb and Jackson emphasize, however, that the successful 
implementation of the CCSSM will depend on the 
willingness of schools and districts to dedicate professional 
development to helping teachers understand and enact 
instruction around key mathematical ideas over time as 
opposed to instruction focused on disparate collections 
of procedures, which has characterized mathematics 
instruction in the past (Stigler & Hiebert, 1999). 

Teacher Preparation to Teach the CCSSM 

Despite the differences between CCSSM and teachers’ 
practices as well as the differences between CCSSM and 
previous state standards (Porter et al., 2011), the percentage 
of teachers feeling well prepared to teach CCSSM vary from 
72 (Cogan, Schmidt, & Houang, 2013) to 93% (Davis, 
Choppin, Roth McDuffie, & Drake, 2013). However, K-12 
teachers working in urban districts feel less prepared to teach 
the CCSSM than teachers who do not work in urban districts 
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(51-63%, N= 800) (Hart Research Associates, 2013). In a 
similar vein, only 50% of teachers working with low 
socioeconomic status students felt very or fairly prepared to 
teach the CCSS compared to 64% of teachers working with 
predominantly high SES students (Hart Research 
Associates). Teachers’ preparedness levels also vary when 
teachers are asked about other student groups such as 
English-language learners (Education Week Research Center 
[EWRC], 2013; Perry et al., 2015). For instance, 29% of 
teachers (NV = 375) surveyed did not feel at all prepared to 
teach the CCSS to English-language learners (EWRC, 2013). 
Overall, teachers feel prepared to teach the CCSS or 
CCSSM, but this level of preparedness varied by school 
location as well as when the focus of that instruction involves 
particular student groups such as English-language learners. 
Previous research has found that teachers perceive that the 
CCSSM has had a number of effects on their instruction that 
is aligned with reform-oriented principles (National Council 
of Teachers of Mathematics [NCTM], 2014). For instance, 
the majority of MSMTs stated that the CCSSM would cause 
them to teach more conceptually, incorporate more 
opportunities for students to explore mathematical ideas, 
solve more complex problems, and emphasize more 
mathematical communication with their students (Choppin, 
Davis, Drake, & Roth McDuffie, 2013; Davis et al., 2013). 

In 2011, as Common Core implementation was just 
underway, teachers identified a need for professional 
development to prepare for the CCSSM (Cogan et al., 2013). 
During the five years since the adoption of the CCSS, 
teachers have received professional development around 
these standards (EPE Research Center, 2013). For instance, 
in a survey of 591 classroom teachers in 2013, 24% of 
respondents had spent two-three days, 17% of respondents 
had spent four—five days, and 41% of respondents had spent 
more than five days on professional development connected 
to the CCSS. The topics and percentages of teachers 
experiencing that professional development were as 
follows: CCSSM (55%); alignment between the CCSS and 
previous state standards (53%); curriculum resources and 
materials (42%); CCSS-aligned assessments (23%); and 
teaching CCSS to specific student groups (15%). A recurring 
theme across these surveys is that while teachers have 
received professional development, about a third would like 
additional professional development in order to feel better 
prepared to teach the CCSS (Choppin et al., 2013; Cogan 
et al., 2013; EPE Research Center, 2013). 

Teacher Perceptions of CCSSM Assessment and 
Evaluation 

Overall, teachers reported a lack of familiarity with the 

Common Core-aligned assessments (EPE Research Center, 
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2013; EWRC, 2013; Scholastic, 2013, 2014). Accordingly, 
teachers expressed an interest in professional development 
focusing on the content of these assessments (EPE 
Research Center, 2013; Perry et al., 2015; Scholastic, 
2013). Despite this lack of information teachers professed 
faith that these assessments will measure the CCSSM 
content standards and SMPs and stated that the Common 
Core-aligned assessments will influence their own 
instructional practices and classroom assessments (Choppin 
et al., 2013). 

As mentioned earlier, the impact of the CCSSM is 
magnified due to the connection of student achievement to 
teacher evaluation. Teachers were aware of this increased 
emphasis of student test scores on teacher evaluation and 
that majority of teachers state that these policies have 
influenced their instructional decisions and assessment 
practices (Choppin et al., 2013). Indeed, in 2015, a total of 
17 states included student growth as a preponderant 
criterion in evaluating teachers and 18 other states included 
student growth as a significant criterion in teacher 
evaluations (Doherty & Jacobs, 2015). 

Teachers’ Perceptions of Curriculum Alignment to the 
CCSSM 

Due to the important role that textbooks play in 
mathematics education (Banilower et al., 2013), surveys 
have examined the connection between curricular resources 
and the CCSS or CCSSM. Researchers have found a lack 
of alignment between curriculum materials that are 
advertised as aligned with the CCSSM and the CCSSM 
(Center for the Study of Curriculum, 2015; Polikoff, 2015). 
Teachers also perceived the curriculum resources that they 
currently use as lacking alignment with the CCSS (EPE 
Research Center, 2013; EWRC, 2013; Scholastic, 2013, 
2014) or the CCSSM (Roth McDuffie et al., 2017). 
Moreover, teachers did not feel that districts provide them 
with the resources they needed to align their instruction to 
the CCSSM, nor did they feel that their districts have an 
effective process in place to determine whether 
supplemental curriculum materials are needed or changes 
need to be made to supplemental materials (Perry et al., 
2015). Additionally, teachers did not feel that their 
curriculum materials enable them to help students with 
disabilities learn the content and practices embedded in the 
CCSSM (Perry et al., 2015). 

The survey at the center of this study builds on previous 
national surveys conducted by other groups (e.g., Education 
Week) as well as those conducted by our group (e.g., 
Davis, Choppin, Drake, & McDuffie, 2014) by focusing on 
the CCSSM, examining other components related to 
MSMTs’ instructional practices, and by presenting results 
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from a larger sample size. In contrast, most surveys on 
teachers’ perceptions have focused on the CCSS in general 
and have included teachers working across grades K-12. 
The survey on which we are reporting in this article focuses 
on MSMTs and provides more current insights into this 
important group’s perceptions with regard to the CCSSM 
and their instructional environment. Specifically, this study 
was designed to answer the following research questions: 

(1) What is the structure underpinning MSMTs’ 
perceptions of the CCSSM and its impact on the 
instructional environment? 

(2) How does a sample of MSMTs perceive the CCSSM 
and the instructional environment given the identified 
structure? 


Methods 

Sample 

Market Data Retrieval (MDR) administered the survey 
nationally during May—June 2015. A total of 1,241 teachers 
working in 39 out of the 42 CCSSM states completed the 
survey. These teachers taught middle school students in 
grades 6, 7, or 8 and were predominantly female 
(Male = 304 [24.5%], Female=935 [75.3%]). The 
majority of teachers considered mathematics to be their 
primary teaching certification/responsibility (N= 1,086 
[87.5%]). The percentages of MSMTs working at different 
types of schools are as follows: Urban (18%), Rural 
(30.3%), Suburban (39.5%), and Town (12.2%). Nearly 
half of the MSMTs in our sample taught in schools that 
were categorized as having 50% or more of their students 
received free or reduced-price lunch. It is important to note 
that while we cannot claim a representative sample on a 
national level, the sample does contain characteristics that 
are similar to MSMTs in the United States. For instance, the 
majority of MSMTs in the United States are female 
(Feistritzer, 2011). The percentage of elementary and 
secondary schools in the United States in 2010-2011 that 
were categorized as urban, suburban, town, and rural were 
26%, 27%, 14%, and 32%, respectively (Kena et al., 2015). 
The percentage of teachers teaching at town and rural 
schools was similar to national percentages while our 
sample of teachers working at urban and suburban schools 
was different. Urban teachers were underrepresented in our 
survey (26% vs. 18%) while suburban teachers were 
overrepresented (27% vs. 39.5%). The percentages of 
MSMTs sampled working at schools with different 
percentages of students eligible for free or reduced-price 
lunches were similar to the percentages of students at U.S. 
public schools. For instance, nationally 44% of public 
schools have 0-50% of their students receiving free or 
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reduced-price lunch (Kena et al., 2015), as compared with 
49.4% in our sample. 
Survey Design 

Given the analyses conducted by Reys et al. (2012) 
regarding the differences between previous state curriculum 
frameworks and the CCSSM, we inserted questions in the 
survey to examine teachers’ perceptions involving CCSSM 
content at the middle school level. The specific inclusion of 
rigor in the CCSSM (McCallum, 2012) led us to include 
questions in the survey to examine teachers’ perceptions of 
rigor in the CCSSM. We built on previous surveys (e.g., 
Cogan et al., 2013; EPE Research Center, 2013) by 
including questions involving teachers’ familiarity with and 
preparedness to teach the CCSSM. With the differences 
between CCSSM and previous state curriculum frameworks 
(Porter et al., 2011; Schmidt & Houang, 2012), we were 
interested in whether teachers perceived that these new 
standards, especially components of the SMPs, had indeed 
impacted their instruction. Given the important role that 
district-adopted curricular resources play in classroom 
instruction (e.g., Tarr et al., 2008), we were interested in 
teachers’ perceptions of the connections between their 
curricular resources and the CCSSM. Interviews with 
MSMTs suggested that assessment and evaluation associated 
with the CCSSM were concerns of teachers (Roth McDuffie 
et al., 2017). These findings resulted in the inclusion of a set 
of questions examining teachers’ perceptions about CCSSM- 
related assessment and the connection of teacher evaluation 
to student achievement on CCSSM-related assessments. 

The increase in digital book sales as well as the fact that 
many states have changed their definitions of textbooks to 
include digital resources (Fletcher, Schaffhauser, & Levin, 
2012) led us to include questions asking teachers about 
their use of digital textbook resources that were part of their 
primary district-adopted textbooks and those that were not. 
The important role that teacher planning plays in the 
classroom environment and students’ learning opportunities 
led to the inclusion of questions related to planning in the 
survey (Clark & Peterson, 1986; Stigler & Hiebert, 1999). 
We were interested in the role of the CCSSM in planning as 
well as teachers’ use of student work in their planning as 
the latter has been found to be an important component of 
professional learning communities that lead to positive 
influence on teachers and students (Fulton & Britton, 
2011). Altogether this survey was designed to measure 
teachers’ perceptions of the CCSSM and its impact on the 
instructional environment. 

Analysis 

Descriptive analyses involved the enumeration of different 

categories and calculations of percentages on Likert-type 
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questions. The percentages of MSMTs strongly agreeing or 
agreeing were combined for reporting purposes. The 
percentages of MSMTs choosing the categories of 
moderately or extensively in questions involving digital/ 
electronic resources were combined and reported. Questions 
involving MSMTs’ use of curriculum materials in planning 
contained five different categories. We decided that once or 
twice a week and almost every day represented frequent use 
of these documents for planning purposes and collapsed 
these two categories into one and reported these percentages. 

Field (2009) describes three purposes for conducting a 
factor analysis: (a) to reduce the size of a data set; (b) to 
develop a survey to measure a variable underlying the data; 
and (c) to identify the structure of one or more variables. As 
described earlier, the purpose of the factor analysis 
conducted for this study is to identify the structure of 
teachers’ perceptions of the CCSSM and its impact on the 
instructional environment. As an initial step in conducting a 
principal components analysis (PCA) the correlation matrix 
among 51 variables was produced. Two variables had no 
correlations of at least absolute value .3 or greater. These 
two variables were eliminated and two other questions were 
removed due to their similarity to other survey questions 
leaving 47 variables included in the PCA. Given 
the connection of standards to teachers’ instructional 
environments, we suspected that factors were correlated 
with one another, so we used oblique rotation (oblimin). A 
Scree Plot is preferable to Kaiser’s criterion for determining 
the number of factors to extract (Field, 2009). The 
inflection point in the Scree plot suggested the presence of 
between five and eleven factors underlying the data set. We 
examined extractions beginning at five factors and steadily 
increased the number of factors extracted until we had well- 
defined factors that explained more than 50% of the 
variability of the data. These criteria were met with eight 
factors extracted. The Kaiser-Meyer-Olkin measure verified 
the sampling adequacy of the analysis, KMO = .853. In 
addition, all of the KMO values were above .7, above the 
accepted limit of .5 (Field, 2009). Bartlett’s test of 
sphericity 7?(1,081)=25,386.71, p< .001 suggested 
that correlations between items was of sufficient size to 
permit PCA. These eight factors accounted for 55% of the 
variability of the data. Cronbach’s Alpha was used to 
measure the reliability of each of the eight factors. The 
reliability of the factors was considered to be good as they 
ranged from .70 to .92 (Field, 2009). 


Results 
The factor analysis appears first and is organized by 
presenting each factor in decreasing level of variability 
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Table 1 
Perceptions Surrounding Professional Support 


Survey Item 


You are familiar with the SMPs described in the CCSSM. 


You are familiar with the mathematical content standards in the CCSSM for the grade levels 


you teach. 


You are familiar with Mathematical Content Standards in the CCSSM for grade level(s) 


other than the grade level(s) you are currently teaching. 
You are well prepared to teach with the CCSSM. 


Your district has encouraged or supported you to try new instructional approaches as a result 


of the implementation of the CCSSM. 


Your district has provided you with adequate professional development around the CCSSM. 
You have collaborated with teachers from other districts to better understand how to 


supplement the CCSSM. 


Percentage Agree or Strongly Agree 


94.5% 
97.2% 


81.5% 


79.9% 
85.4% 


63.5% 
50.6% 





explained. The percentage of MSMTs agreeing or strongly 
agreeing with each of the survey items associated with a 
particular factor appears within that factor’s description. 
Factor Analysis 

The factor loadings after rotation appear in the Appendix. 
The first factor, Professional Support, consisted of seven 
survey items and explained 16.0% of the variance in the 
sample. This factor illustrates the many tendrils emanating 
from support that extend into other areas associated with 
the CCSSM and teachers’ instructional environments. For 
instance, professional development was connected to 
teachers’ familiarity with the CCSSM process and content 
standards as well as their perceptions of their preparedness 
with respect to the CCSSM. This factor also included 
teachers’ collaborations with their colleagues from other 
districts as well as their perceptions of their district’s 
support for new instructional approaches. 

The descriptives for this factor appear in Table 1. 
Although MSMTs are familiar with CCSSM content and 
SMPs, a somewhat lower percentage of MSMTs felt well 
prepared to teach the CCSSM. Table | also shows that just 
barely half of the MSMTs surveyed collaborated with 
teachers from other districts in understanding the CCSSM. 
The majority of MSMTs report that their school districts 
have encouraged new instructional approaches. One-third 
of MSMTs surveyed noted that they had not received 
adequate professional development around the CCSSM. 


Table 2 


The second factor, Teachers’ Use and Reading of Non- 
District-Adopted Digital/Electronic Resources, extracted 
from the data consisted of five survey items and accounted 
for 9.3% of survey variability. The majority of items in this 
category were composed of teachers’ perceptions of their 
reading and use of digital/electronic resources from 
resources that were not connected with their district- 
adopted curriculum resources. Interestingly, these resources 
were not connected to any of the survey items associated 
with teachers’ district-adopted curricula. As seen in Table 2 
MSMTs were more likely to use these digital resources for 
remediation than inquiry and they were more likely to 
consult the teacher resource versions of these resources than 
the resources intended strictly for students. 

The third factor, Influence of CCSSM Assessment and 
Teacher Evaluation on the Classroom Environment, 
consisted of five survey items and explained 7.1% of the 
variance. These items consisted of teachers’ perceptions of 
the influence of CCSSM-aligned assessment and teacher 
evaluation on instructional decisions, assessment practices 
and content/practices emphasized during classroom 
instruction. In addition, an item connecting CCSSM-related 
assessment and teacher evaluation was a component of 
this factor. The appearance of assessment and teacher 
evaluation as one of the top factors speaks to the 
importance of these CCSSM-related policy components for 
teachers. 


Teachers’ Use and Reading of Non-District-Adopted Digital/Electronic Resources 


Item 


Moderate/Extensive Use 





Read digital/electronic student text pages or worksheets 
Read digital/electronic teacher resources 


Use digital/electronic resources to plan primary activities for your lessons mn 
Use digital/electronic resources to create supplemental activities for practice or remediation 
Use digital/electronic resources to create inquiry-based activities 


50.6% 
63.1% 
64.5% 
70.7% 
57.2% 
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Table 3 


Influence of CCSSM Assessment and Teacher Evaluation on the Classroom Environment 


Item 


Percentage Agree or Strongly Agree 





Increased emphasis on student test scores in the evaluation of teachers. Wh 
The teacher evaluation process in your state/district influences your instructional decisions. 
The teacher evaluation process in your state/district influences your assessment practices. 


State assessments will influence your assessment practices. 


State assessments will help you decide what CCSSM mathematics content and practices to 


emphasize. 


71.7% 
69.2% 
65.6% 
77.7% 
66.8% 





Table 3 shows the descriptives associated with the third 
factor. A majority of MSMTs has noticed the increased 
emphasis on student test scores in teacher evaluation. 
Although the MSMTs recognized this emphasis, a slightly 
smaller majority noted the influences of teacher evaluation 
on instructional decisions and assessment practices. The 
results in Table 3 also illustrate the strong effect that the 
CCSSM.-aligned assessments have on teachers’ assessment 
and instructional practices. For example, over three-fourths 
of the MSMTs sampled agreed or strongly agreed that the 
state assessments will influence their own assessment 
practices. 

The fourth factor, Influence of the CCSSM on Classroom 
Instruction vis-a-vis Previous State Standards, extracted 
from the data comprised eight survey questions and 
accounted for 6.3% of the variance in participant survey 
responses. This factor included items that indicated how the 
CCSSM had influenced teachers’ instructional practices. 
These practices consisted of mathematical processes such as 
communication and exploration, content such as concepts 
and procedures, and opportunities for students to struggle in 
mathematics. In addition, this category included change in 
classroom instruction in general as well as the influence of 
the CCSSM on how teachers use classroom materials. 

Table 4 shows the questions associated with MSMTs’ 
perceptions of the influence of the CCSSM on their daily 


Table 4 


instructional practices. MSMTs perceived changes to their 
classroom instruction that are aligned with reform-oriented 
techniques (NCTM, 2014) such as communication and 
exploration. Similar to past surveys of MSMTs’ 
perceptions of the CCSSM (Davis et al., 2013) teachers in 
this sample saw the CCSSM as requiring teachers to teach 
both more conceptually and more procedurally. Three- 
quarters of the surveyed MSMTs stated that the CCSSM 
had caused them to change their classroom instruction. 
More importantly, MSMTs perceived that the CCSSM 
required them to provide more opportunities for students to 
struggle when solving problems. 

The fifth factor, Teachers’ Use and Reading of District- 
Adopted Digital/Electronic Resources, consisted of eight 
survey items and explained 4.6% of the sample variance. 
This factor includes the digital/electronic resources 
associated with the district-adopted curriculum. In addition, 
it also includes elements associated with the district- 
adopted curriculum such as its support for concepts and 
procedural fluency as well as the fact that this curriculum 
was adopted to support implementation of the CCSSM. 

Table 5 shows similarities to MSMTs’ use of digital/ 
electronic resources from digital resources not associated 
with their district-adopted curriculum (Table 2). That is, 
MSMTs were more likely to use digital teacher resources 
than student text pages associated with their curriculum and 


Influence of the CCSSM on Classroom Instruction vis-a-vis Previous State Standards 





Survey Item 


The CCSSM have required you to change your classroom instruction. 
The CCSSM have required you to emphasize mathematical concepts more with your 


students. 


The CCSSM have required you to emphasize mathematical procedures more with your 


students. 


The CCSSM have required you to incorporate more student exploration. 

The CCSSM have required you to emphasize more mathematical communication. 

The CCSSM have required you to emphasize more solving of complex problems. 

The CCSSM have required you to provide more opportunities for students to struggle while 


solving problems. 


The CCSSM required you to change how you use curriculum materials. 


Percentage Agree or Strongly Agree 


75.0% 
77.8% 


64.3% 
TOTS 
85.9% 
89.0% 
79.7% 


77.0% 
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Table 5 
Teachers’ Use and Reading of District-Adopted Digital/Electronic Resources 


Item 


Read digital/electronic student text pages or worksheets 
Read digital/electronic teacher resources 


Use digital/electronic resources to plan primary activities for your lessons 
Use digital/electronic resources to create supplemental activities for practice or remediation 


Use digital/electronic resources to create inquiry-based activities 


Your district adopted a new curriculum program to address the CCSSM. 
The district-adopted curriculum materials that you currently use support your students’ 


development of computational or procedural fluency. 


The district-adopted curriculum materials that you currently use support your students’ 


development of conceptual understanding. 


Moderately or Extensively 


54.3% 
73.2% 
69.0% 
74.5% 
55.8% 
Agree or Strongly Agree 
65.5% 
56.6% 


61.5% 





less likely to use digital resources to create inquiry-based 
activities. The majority of MSMTs sampled worked in 
districts that had adopted a new curriculum to address the 
CCSSM. The majority of MSMTs perceived that their 
district-adopted curriculum materials promoted procedural 
fluency as well as conceptual understanding. 

The sixth factor, Strength of Perceptions of the CCSSM- 
based Assessments, explained 4.4% of the variance in the 
data and consisted of four survey items. MSMTs’ 
perceptions of the CCSSM-aligned assessments appeared as 
a factor separate from their perceptions of the influence of 
the same assessments on various aspects of classroom 
instruction. As seen in Table 6, the majority of MSMTs 
sampled agreed that the CCSSM-aligned assessments will 
measure content appearing in the standards, but were less 
optimistic that these assessments will measure the SMPs or 
contain more complex problems. Nonetheless, MSMTs did 
feel that these externally produced assessments will measure 
content similar to their own classroom assessments. 

The seventh factor, Extent of MSMTs’ Planning 
Practices, explained 3.7% of the variance in the data and 
consisted of six survey items. This category included items 
such as the use of the CCSSM or other documents when 
planning and MSMTs’ planning with colleagues or 
discussion of mathematics content when planning with 
colleagues. Despite the importance of planning in teachers’ 
work (Clark & Peterson, 1986; Stigler & Hiebert, 1999), it 


Table 6 
Strength of Perceptions of the CCSSM-Based Assessments 


Item 


State assessments will be aligned with the CCSSM Content standards. 
State assessments will contain problems involving larger numbers, more decimals, 


and more fractions. 
State assessments will assess each of the eight CCSSM SMPs. 


State assessments will measure content similar to your classroom assessments. 


was surprising that this factor was near the bottom of the 
list of factors associated with MSMTs’ perceptions of the 
CCSSM and its impact on their instructional environment. 

Table 7 displays the descriptives associated with the 
seventh factor. Two-thirds of respondents referred to the 
CCSSM document when planning lessons with fewer 
MSMTs using other documents to make sense of the 
CCSSM. District curriculum frameworks also play an 
important role in teachers’ planning practices. Although 
studying student work can help teachers to plan in ways 
that promote student understanding especially when used 
within professional learning communities (Fulton & 
Britton, 2011), the majority of MSMTs did not engage in 
this activity on a regular basis when planning. 

The last factor titled, Level of Perceived Rigor within the 
CCSSM, encompassed three survey items, explaining 3.4% 
of the variance in the sample. Two questions asked MSMTs 
about the CCSSM content and practice standards and rigor. 
The last question involved new content of the CCSSM. 
Thus, not only are teachers identifying new content for their 
grade level in the CCSSM they are also associating this the 
new content appearing in the CCSSM as rigorous when 
compared to their previous state standards. Despite the 
connection of rigor with new content, this factor was less 
important when compared with others such as the use of 
digital/electronic curriculum materials, explaining the least 
amount of variability in MSMTs’ perceptions. 


Percentage Agree or Strongly Agree 


78.1% 
56.0% 


59.4% 
64.2% 
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Table 7 
Extent of MSMTs’ Planning Practices 


Item 


Regularly study student work when planning. 

You refer to the CCSSM when planning. 

You refer to your district curriculum framework when planning. 
You refer to other documents that explain the CCSSM. 


You regularly discuss mathematics content with colleagues. 
You regularly plan with colleagues. 


Percentage Once or 
Twice a Week/Almost Every Day 


42.5% 
66.6% 
62.0% 
52.9% 
Agree or Strongly Agree 
74.4% 
64.9% 





As seen in Table 8, a strong majority of MSMTs stated 
that both the content and SMPs were more rigorous than 
similar components of their previous state standards. In 
addition, a majority of MSMTs noted that the CCSSM 
contains new content for the grade level at which they teach. 


Discussion 
MSMTs’ Use of Curricular Resources 

Unlike previous research involving teachers’ perceptions 
involving the CCSSM we explored a number of 
components at the intersection between curricular resources 
and the Common Core. Most of the teachers in our sample 
felt well prepared to teach the CCSSM yet around 40% of 
the teachers in our sample did not perceive their curriculum 
resources to align with components of the CCSSM. Indeed, 
the majority of our survey respondents noted that the 
CCSSM had caused them to change how they use their 
curriculum materials. These findings suggest fertile areas of 
research highlighted by the following questions: (a) How 
are teachers using their curricular resources differently due 
to the presence of the CCSSM? (b) What aspects of the 
CCSSM, their curricular resources, and the instructional 
environment have led to different curricular uses? Our 
study suggests that answers to questions such as these can 
help illuminate the nature of CCSSM implementation in 
middle school classrooms. 

Moreover, our research suggests that teachers are 
frequently accessing both district-adopted and non-district- 
adopted digital resources as part of their planning for 
classroom instruction. Our results also indicate that these 


Table 8 
Level of Perceived Rigor Within the CCSSM 


resources are used more often for remediation. Given the vast 
amount of mathematics education resources that exist online 
it is important to understand how teachers make decisions 
regarding these materials and how those resources connect to 
CCSSM-aligned instruction. Additionally, the factor analysis 
shows that MSMTs’ use of digital resources associated with 
their curriculum is separate from their use of digital resources 
that they locate elsewhere; however, the latter factor was 
more important accounting for 9% of survey variation. The 
factor analysis also illustrates that non-district-adopted digital 
resources were not connected to the district-adopted print 
curriculum. This suggests a lack of curricular coherence, 
which may have negative effects on students’ understanding 
of the mathematics represented in the CCSSM, especially as 
progressions within and across grade levels are an important 
component of these standards (McCallum, 2012). In sum, the 
proliferation of easily located, externally produced digital/ 
electronic resources that teachers are using to supplement 
their district-adopted curriculum is an area in which future 
research needs to be conducted. 
MSMTs’ Perceptions of CCSSM-Aligned Assessments 
Survey respondents reported less confidence in state- 
based CCSSM assessments in a number of different areas 
including assessing the CCSSM SMPs or their similarity to 
the MSMTs’ own assessments. This conflicts with previous 
research (Choppin et al., 2013) and may reflect a shift in 
teachers’ perceptions with regard to these assessments. One 
effect of this loss in confidence with regard to these 
assessments is their decreased influence on our survey 
respondents’ classroom practices in terms of the content 


—_—— Ssh 


Survey Item 


The Content Standards are more rigorous than your previous state standards. 
The Practice Standards are more rigorous than your previous state standards. 
The CCSSM represent new content for the grade level you teach. 


Percentage Agree or Strongly Agree 


85.5% 
87.8% 
82.5% 
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and practices they will emphasize on a daily basis. Many 
components such as communication and exploration are 
found in the SMPs. If MSMTs feel that the SMPs will not 
be assessed, they may be less likely to incorporate them in 
their daily practices. Research is needed to ascertain 
teachers’ confidence that these assessments will align with 
all of the CCSSM (including the SMPs), how these 
perceptions will influence their classroom instruction, and 
how these changes will influence student understanding of 
content and the SMPs. 

MSMTs’ Perceptions of CCSSM Professional 
Development 

Professional support explained the greatest percentage of 
variability among survey responses. The findings from the 
factor analysis suggest that professional development is 
linked with a number of important components of the 
CCSSM. For instance, professional development was 
associated with familiarity with the content standards and the 
SMPs, overall preparedness associated with the CCSSM, 
encouragement of new instructional approaches, and 
collaboration with teachers in other districts. This finding also 
illustrates the power of CCSSM professional development. 
That is, CCSSM professional development is associated with 
a number of different components that can assist with the 
implementation of these standards. Nonetheless, one-third of 
our respondents agreed that they had received inadequate 
professional development with regard to the CCSSM. This 
finding is similar to what previous surveys have found (e.g., 
EWRC, 2013). Despite experience implementing the 
CCSSM and receiving professional development with regard 
to it, teachers are still in need of professional development. 
Indeed, this matches what we know about professional 
development; it must continue over time (Clarke, 1994; 
Elmore, 2002; Hawley & Valli, 1999). 

Our study also found that while teachers are very familiar 
with the CCSSM content standards for the grade levels that 
they teach, they are less familiar with the content standards 
for grade levels other than their own. This suggests one area 
in which teachers could receive: CCSSM-based professional 
development. Given that one of the pillars of the CCSSM is 
coherence both from a mathematical logical perspective as 
well as a cognitive development perspective (McCallum, 
2012), professional development focusing on content 
standards in other grades could also promote the 
progressions across grade levels in different content strands 
such as number and operations. The factor analysis 
suggests that promoting teacher collaborations across 
school buildings or districts is linked with preparedness and 
familiarity to teach the CCSSM. Thus, promoting these 
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types of focused collaborations may be a viable form of 
CCSSM professional development. 
Instructional Changes Due to the CCSSM 

The results of our study indicate that MSMTs perceive that 
their instruction has changed due to the CCSSM. With regard 
to how that classroom instruction changed, survey results 
indicate that teachers are emphasizing both conceptual 
understanding as well as computational/procedural fluency. 
Given the historical emphasis on procedures in USS. 
mathematics education classrooms (e.g., Stigler & Hiebert, 
1999) this is indeed a hopeful sign that classroom instruction 
in the United States may be moving closer to the vision of 
mathematics education held by professional organizations 
(NCTM, 2014). Given the connection between student 
struggle and student achievement (e.g., Hiebert & Grouws, 
2007), it is encouraging to see that the teachers in our sample 
perceived that the CCSSM provides more opportunities for 
students to struggle when attempting to solve problems in 
mathematics. Further, the factor analysis highlights that a 
number of changes associated with processes such as 
communication and exploration (NCTM) were connected 
with changes to classroom instruction as well as changes to 
how MSMTs use their curriculum materials. The finding that 
this factor explained the greatest variation in survey responses 
speaks to the potentially transformative role of the CCSSM. 
Nonetheless, these results suggest the need for further research 
to examine how these purported changes are reflected in the 
enacted curriculum. That is, teachers may simply be reporting 
on the survey what they think researchers want to hear. 


Conclusions 

This study moves beyond previous studies by focusing 
on MSMTs’ perceptions of the CCSSM and its influence on 
components of the instructional environment such as 
curriculum resources in both print and digital forms. On the 
one hand, the results suggest that teachers may be using 
digital resources in ways that do not align with their district- 
adopted resources. On the other hand, the CCSSM appears 
to be influencing MSMTs’ classroom instruction in more 
reform-oriented ways. The results of this study also suggest 
several fertile areas of research such as locating the factors 
that influence MSMTs’ use of online digital resources and 
understanding the specifics regarding how the CCSSM 
changes teachers’ enacted curricula. These are critical areas 
we need to understand in order to discern the impact of 
CCSSM at the middle grades. 
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Appendix 
Summary of Factor Analysis (N = 1,241) 
Rotated Factor Loadings 

Survey Item 1 2 3 4 D 6 7 8 
Familiar with practices .80 = () 7, .03 —.04 09 4 = )S .08 
Familiar with content standards 79 ents .00 .04 10 .00 —.08 14 
Familiar with content other grades 76 =—00 | 10 by =i OT .03 
Well prepared to teach CCSSM 76 Ol —.06 2.08 ==109 02 01 =.02 
Adequate professional development 53 10 02 01 i116 .02 .20 3.05 
Collaboration with other districts 51 = 1) 04 07. = 0S = OL 09 —.08 
New instructional approaches 49 .07 —.04 =. —.08 .06 .16 01 
Use DN* resources to plan .06 —.84 —.08 —.06 res .08 07 .00 
Use DN resources for remediation .10 —.82 =—412 —.09 «10 .08 .03 #08 
Use DN resources for inquiry 09 —.78 =n 0 =, 09 = 16 .04 .07 05 
Read DN teacher resources .03 15 —.04 aU = 18 .03 07 =03 
Read student-based DN resources OE —.68 seta 01 aD} =—iUa .08 = 01 
Teacher evaluation influences instruction .04 Ut 12 =09 —.08 .00 — .06 —.06 
Teacher evaluation influence classroom assessment .08 ah? 71 92.05 —.08 01 —.08 —.01 
State assessments influence classroom assessment 01 .00 .67 = 06 .00 .09 .00 = 
Emphasis of assess on evaluation ely =nU 52 05 O01 ms .00 .04 
State assess determine content/practices IS .02 51 aT .O7 27 .09 01 
Emphasize more communication “43 O07 202 — .83 05 .03 7:10 —h2. 
Emphasize more exploration ihe 206 01 wat = 02 02 so (KZ, =e 
Emphasize more concepts 04 mee 02 = 310 = Ud, ale Faget bi .04 
Emphasize complex problems .04 .00 01 =e —.03 att — .06 ‘ES 
More opportunities for student struggle .05 04 01 —.68 .02 .03 = 06 .08 
Change classroom instruction 113, 08 .06 e~ 58 .06 —.04 .10 15 
Emphasize math procedures as .04 04 = O08 = ()5 = 0 .0S .02 
Change how you use curriculum —.08 ae .08 345 .09 =e To ty 
Read DAS teacher resources 243 —.18 09 =.01 S97 .03 —.08 = 01 
Use DA to plan .03 —.24 .08 09 eradil —05 77017 .08 
Read DA versions of student text ll el GY .03 02 —.74 — (05 .02 02 
Use DA for remediation .05 —.24 .05 .08 st ALE Oe .04 
Use DA for inquiry .09 Dh .04 .00 —.66 —.06 = .00 
Curriculum supports concepts 12 Al srubil f ht =z pbt .20 =. 05 
New curriculum for the CCSSM .05 B33 = .06 —.14 —.48 .06 ahs .06 
Curriculum supports procedural fluency 14 40 =U =) —.44 04 Le Sue 
State assessment aligned with class assess .00 —.06 rin .00 .00 2 .04 arte. 
State assess measure SMPs #307 —.06 “2 05 02 68 .03 10 
State assess complex problems ==,Q3 +01 04 Ql 04 .67 .03 +03 
State assess aligned with content 01 —.04 .20 .08 =) .67 .03 .06 
Plan with colleagues 04 are aU aOt .06 14 84 =.03 
Discuss content when planning 12 206 a1 205 .03 18 .79 .00 
Study student work when planning el I —.05 —.09 O01 7.03 02 74 —.04 
Use district framework to plan re (7 05 .16 O01 ==) 7. = 19 41 .03 
Refer to other docs when planning —.04 ee 28 01 U3 ars 37 ahd 
Use the CCSSM when planning ; .09 09 31 .00 =O] salto 35 02 
Rigor in CCSSM content standard .03 09 —.07 mieli2 a7 01 —.04 84 
CCSSM represent new content .06 02 aS .06 .00 .08 .05 AD 
CCSSM SMPs are more rigorous 07 05 = U3 = 13 = UL .O1 = U9 19 
Eigenvalues 4.50 4.25 3.08 4.97 4.79 2.49 a0 2.69 
% of variance 16.1 9.3 Tel 6.3 4.6 4.4 a9 3.4 
o 81 92 70 84 83 ohn 70 9 


*This refers to digital resources that do not accompany the district-adopted textbook. 
These include digital resources that accompany the district-adopted textbook. 
Note. Factor loadings of at least .4 appear in bold. 
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This study utilized the Draw-A-Scientist Test Checklist (DAST-C) to assess the illustrations of scientists in the most 
recent three years of NSTA Recommends book lists. A total of 15,778 images were contained in the 148 books from those 
lists, of which 1,676 were of scientists. ANOVA procedures revealed no significant differences in stereotypical elements 
across the three years of books. However, three notable stereotypical elements were present in large percentages in books 


from all years: predominance of male images, non-minority scientists, and scientists who were not youthful. 


Literature is often a bridge into the sometimes difficult 
and intimidating world of teaching science because some 
elementary teachers are more comfortable with language 
arts than science (Kazempour, 2014). The use of trade 
books and picture books to integrate literacy into science 
instruction time is commonly used as a means of 
maximizing students’ understanding of specific content- 
related concepts. One of those concepts encompasses what 
scientists look like, who they are, and what they do—what 
is called “perceptions of scientists.” However, there is 
concern that some books containing illustrations of 
scientists can actually create or increase misconceptions 
about scientists. Helping teachers make good choices about 
what picture books they introduce to their students may 
impact students’ understanding about where scientists work 
and what they do. It may also influence some children in 
terms of interest and engagement in career choices (Archer 
et al., 2010; Shope, 2006). Those choices can be influenced 
by students’ perceptions about scientists. The perceptions 
of scientists students hold may relate to their attitudes 
toward science and scientists (Finson, 2003; Fung, 2002). 
O’Brien, Kopala, and Martinez-Pons (1999) linked self- 
efficacy to the probability of an individual choosing a 
career in that field. It also has implications regarding the 
formation of science identity in children, as noted in the 
work reported by Farland-Smith (2012). Science identity 
includes one’s perception that he/she can be a successful 
investigator in science contexts and is viewed by others as 
being competent in science (Brickhouse & Potter, 2001; 
Carlone, 1994; Farland-Smith, 2012). From the above, one 
could reasonably conclude that individuals who hold 
negative perceptions of science or scientists may be 
unlikely to pursue science courses or select science as a 
career. One way to address these issues is through the use 
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of science trade books and picture books in instruction 
(Farland, 2006a,b; Farland-Smith, 2012). 

Embedded within perceptions of scientists concerns are 
issues concerning the portrayal of scientists in terms of 
what they do and how they do it. In brief, how science is 
represented as a human endeavor has been consistently 
addressed in early science education reform documents 
and through more recent science reform efforts. Science as 
a human endeavor is a significant component of both the 
AAAS (1993) Benchmarks for Scientific Literacy (Project 
2061) and the National Science Education Standards 
(NRC, 1996). The NSES delineated how science as a 
human endeavor was evidenced by three factors: (1) 
people are engaged in science and technology for a long 
time, (2) those who have contributed throughout history to 
the knowledge base have been both men and women and 
of various ethnicities-some of whom make careers in the 
sciences, and (3) science is an ongoing endeavor that does 
not end. This advocacy about the human side of science 
has continued in the most recent NRC (2012) standards— 
A Framework for K-12 Science Education. The 
Next Generation Science Standards (NGSS Lead States, 
2013) explain the importance of K-12 students coming to 
know how scientists have investigated and answered 
fundamental questions in addressing major challenges 
facing society. The standards delve in depth about this 
human face of science through Dimension 1 (Scientific 
and Engineering Practices). It would seem logical that 
science educators would, therefore, look for ways to help 
students see science as a human endeavor. There are, of 
course, multiple ways this can be addressed. One of them 
is through the use of trade books and picture books that 
children can relate to and enjoy reading (Segun, 1988; 
Tapscott, 2009). While both text and illustrations function 
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together to promote comprehension (Hibbing & Rankin- 
Erickson, 2003; Martinez & Harmon, 2012; Verhallen & 
Bus, 2011), pictures can also help readers relate to 
characters when the illustrations are reflective of the 
readers themselves (Hefflin & Barksdale-Ladd, 2001; 
Roethler, 1998). Furthermore, books that contain 
stereotypical illustrations can be problematic (Mendoza & 
Reese, 2001), but those in which students can see 
themselves, or relate to, are viewed as positive and 
motivational (Al-Hazza, 2010). Additionally, there is 
some evidence that young children can visualize 
themselves in a science-related role via pictures in a 
children’s book (Pantoya, Aguirre-Munoz, & Hunt, 
2015). Therefore, illustrations of scientists can be 
important when engaging students in visualizing science 
as a human endeavor. 

The purpose of this study was to investigate the 
illustrations, images, or photographs of scientists portrayed 
in picture books promoted for use in elementary and middle 
school classrooms. The focus was not on the stories being 
told in the books, but on what the scientists looked like and 
what they were doing. The books examined in the study 
were those that have been promoted by the National 
Science Teachers Association through its NSTA 
Recommends book lists. By their inclusion in these lists, 
books are implicitly viewed as being well vetted and of 
high quality with regard to their use in classrooms. 


Theoretical Framework 

Typically, an individual will become more likely to 
engage in science and view themselves as someone who 
does science (e.g., a scientist) if he/she can match his/her 
perceptions of what a scientist is like to his/her perception 
of self. This is particularly true for children and adolescents 
and is referred to as self-to-prototype matching theory 
(Rawson & McCool, 2014). The self-to-prototype matching 
also has implications with regard to one’s science identity, 
which is essentially how one views self as being able to 
participate successfully in science and how one is viewed 
by others as being successful in science (Archer et al., 
2010; Farland-Smith, 2012). By the time children reach 
adolescence, they have well-developed perceptions of what 
scientists are and their own science identities, and those 
perceptions usually run counter to what science educators 
desire them to be. This has long-term consequences with 
regard to adolescents’ choices of taking additional 
coursework in science, pursuing science-related hobbies, 
and later choosing careers in science and science-related 
fields (Hannover & Kessels, 2004). 
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There are a number of potential sources of influence that 
impact learners’ development with respect to their science 
identities and perceptions of scientists. Such sources could 
include role models, parental influences, school influences, 
and media (Farland-Smith, Finson, Boone, & Yale, 2014; 
Steinke et al., 2007). Of these, media were found to have 
some of the most significant impacts on children’s science 
identities. Media consist of multiple things, including 
television, movies, the Internet, and books. A common 
medium utilized by parents and teachers is trade books. 
Trade books, by definition, are books published for the 
general public and usually convey information in a story- 
type format, unlike textbooks that present factual or 
descriptive information and are usually not available to the 
public at large. Trade books, particularly picture books, are 
often the first exposure children have to what scientists are 
like (Rawson & McCool, 2014). If constructed well, trade 
books can provide students with images of scientists and 
what scientists do—to which the students can relate and 
constructively build their self-to-prototype schema. Poorly 
constructed trade books may very likely convey 
stereotypical images of scientists that fail to help learners 
develop positive self-to-prototype perceptions and positive 
science identities. An example would be having minority 
students and/or females read a trade book showing a middle- 
aged male Caucasian scientist—a representation that those 
students cannot easily relate to. Good, Woodzicka, and 
Wingfield (2010) noted that such disconnects can adversely 
impact students’ academic performance. Unfortunately, the 
predominance of Caucasian males in trade book illustrations 
has continued (Delgato, 2009; Elgar, 2004; Kahveci, 2010; 
King & Domin, 2007), and balance in representations of 
gender and minority figures remains unattained (Rawson & 
McCool, 2014). This state of affairs has considerable 
implications for science education and the populating of 
science careers in the future. 

From the foregoing, it is reasonable to conclude that the 
teacher’s selection of trade books for use in the classroom 
becomes an important consideration with regard to 
students’ formation of science identities and _ self-to- 
prototype conceptualization. To help in this endeavor, some 
organizations such as the National Science Teachers 
Association have undertaken the task of vetting science 
trade books and generating lists of them to recommend to 
teachers for their use. The obvious assumption is that if 
such organizations have vetted the trade books, then 
teachers should not be concerned about the way those 
books portray scientists or the work they do. However, this 
assumption may be erroneous and is worthy of closer 
examination. 


251 


DAST-C and NSTA Recommends Books 


Background 
Sources of Children’s Perceptions of Scientists 

Exactly where children derive their perceptions of 
scientists has been a question not easily resolved. In their 
2014 study, Farland-Smith et al. identified a variety of 
potential sources of influence that could contribute to 
children’s perceptions of scientists. Among those influences 
were media (such as movies, television, and internet); 
books (including picture books, comic books, etc.); parents 
(e.g., imparting of values, informal activities); and schools 
(e.g., activities that occur there). The researchers had 206 
elementary students sketch multiple pictures of scientists, 
and then conducted interviews with the children to ascertain 
what the students were drawing from when creating their 
illustrations. Media sources were found to contribute to 
students’ perceptions of scientists, although each of the 
sources had some degree of influence. 

In 2006, Farland reported that some third-graders’ 
perceptions of scientists were affected through weekly 
readings of nonfiction, historical-based trade books that 
depicted the work of scientists. Bazler and Simonis (1990) 
noted the messages delivered through textbooks showing 
scientists were clearly that science was a male endeavor and 
accomplishments of females were missing, trivialized, or 
even criticized. When females were included in the books, 
they were presented as having passive roles. Further, the 
books contained a dearth of topics that would be of interest 
to females. 

Importance of Images 

Quality illustrations in picture books include colorful 
images students will want to look at and be able to refer to 
over and again (Verhallen & Bus, 2011; Xiung, 2009). 
Analysis of images usually differentiates between 
photographs and other types of illustrations, and considers 
their attributes, qualities, and appropriateness for use with 
targeted student age groups. Children have particular 
preferences for images that help them comprehend the 
content being presented in books (Fang, 1996; Glenberg & 
Langston, 1992), and this becomes important to consider 
when books are being selected to help teach concepts, even 
if those concepts are indirectly taught, such as when 
attributes of scientists are included but are not the primary 
construct intended to be conveyed by the illustrations. 

Included in the extant research regarding the impact of 
illustrations and photographic images on elementary 
children were examinations of attributes such as color (bold 
and bright versus muted and pastels), realistic versus 
conventionalized presentation, sharp versus rounded 
lines, line drawings versus drawn images, and drawings/ 
paintings versus photographs. Rationales for considering 
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the importance of such elements encompass improvement 
of reader comprehension and development of specific 
language. The overall design of visual features and 
illustrations typically guide the reader in comprehending 
and linking elements of stories (Andrews, Scharff, & 
Moses, 2002; Wolfenbarger & Sipe, 2007). Appropriate 
and well-done illustrations help children develop a 
language of science (beyond simple vocabulary), extending 
to the language of inquiry: observation, logically derived 
hypothesizing, question-posing, and examination of 
evidence (Pappas, 2006). Gains in these understandings of 
the language of science can be helpful to children as they 
develop their perceptions of what scientists do and how 
they conduct science investigations. 

Additionally, how children see themselves in picture 
book illustrations is important to developing an interest in 
reading. Hefflin and Barksdale-Ladd (2001) interviewed 
African American adult and children readers, who 
expressed desires to see themselves in what they were 
reading. It was only when the books included characters 
with whom they could relate that readers began to feel 
connected to books and developed more interest in reading. 
Conversely, characters presented in books can also be 
viewed by nonminority readers as conveying clear messages 
regarding who can fulfill the roles characterized in books. 
As an example, children’s book author Myers (2014) 
recounted an interview involving two equally qualified 
applicants for a chemist’s job. Myers’s colleague had to be 
convinced to send the African American applicant to the lab 
for an interview because the colleague could not believe the 
minority applicant was actually a chemist, despite his 
résumé. Myers alluded to the idea that the colleague’s 
perceptions about who could be a chemist were derived, in 
part, from what the colleague had read in children’s books 
while growing up, and those books simply did not show 
real scientists being from minority populations. Myers’s 
conclusion was that children’s books have a powerful role 
in promoting such roles for both minority and majority 
people, and that these notions carry forward into adulthood. 
Content Analysis 

The books that were analyzed in this study were the 
books in the National Science Teachers Association’s 
NSTA Recommends book lists for 2014, 2015, and 2016. 
Children’s picture books are books that not only tell a story, 
but give the reader an experience that is visual in nature, 
and is a book that “displays respect for children’s 
understandings, abilities and appreciations” (ALSC, 2016). 
Children’s chapter books are those that rely on the written 
text to convey a story or information. Trade books are 
defined by Durkin (1974) as books written for use in 
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Table 1 
Number of Books on NSTA Recommends List 








2014 2015 2016 

# on NSTA #Assessed % Assessed #onNSTA #Assessed % Assessed #onNSTA #Assessed % Assessed 
List List List 

aS 16 29.09 49 22 44.89 44 18 40.90 





libraries or bookstores rather than being utilized as 
textbooks. Need a sentence or two here about how NSTA 
selects the books for NSTA Recommends (Donna?) The 
specific targeted group of books from the NSTA 
Recommends lists were picture trade books that contained 
illustrations of scientists. 

The specific number of books on the NSTA Recommends 
list varied across the three targeted years, as shown in Table 
1. As with the number of images in the books across the 
three years, the actual number of images of scientists also 
varied (see Table 2), ranging from a low of 4.18% in the 
2016 books to a high of 18.34% in 2015. From these data, 
no particular trend in the number of images of scientists 
included in books could be identified. Some of the books 
showed scientists that could clearly be categorized as 
working in earth, life, or physical science. A total of 57 of 
the books could be classified as having orientation toward 
one of these three disciplines. The majority of images that 
could be classified in this way were of life and biological 
sciences (nearly three times the other disciplines), followed 
next by earth and then by physical science (see Table 3). 


Instrumentation 

The instrument utilized in this study was the Draw-A- 
Scientist-Test Checklist (DAST-C). The DAST-C was 
constructed using elements identified from a number of 
past studies on perceptions of scientists going as far back as 
the 1950s, beginning with the classic Mead and Metraux 
study in 1957, and then followed by Chambers (1983). 
Mead and Metraux (1957) asked 35,000 high school 
students to write essays in which they described what they 
thought scientists looked like. The perception that emerged 
from the essays was that scientists were elderly or middle- 


Table 2 

Number of Images in NSTA Recommends Books 

2014 2015 

# of # of Scientist %ofImages # of Images # of 

Images Images that are Scientist 
Scientists Images 

6,591 677 10.27 4,361 800 


aged males wearing white lab coats and eyeglasses, 
working in dangerous laboratory settings. Chambers (1983) 
built from that work to develop the DAST, differing from 
the work done by Mead and Metraux in that students were 
not asked to write anything but were asked to make 
drawings. His analysis of drawings brought to light that 
seven specific attributes existed that were persistent across 
drawings. Those attributes were men wearing lab coats and 
eyeglasses, having facial hair (like beards or mustaches), 
surrounded by symbols of research (e.g., scientific 
instruments), symbols of knowledge (e.g., books, periodic 
tables, filing cabinets), technology (e.g., rockets, 
computers), and relevant captions (e.g., “eureka!” or 
formulae). These attributes have become recognized as the 
elements comprising the “stereotypical” scientist. 

In their 1995 study, Finson, Beaver, and Cramond built 
upon Chambers’s work to develop a_ checklist-type 
instrument that could help quantify the attributes arising in 
drawings done by children about scientists. They called 
their instrument the DAST-C. The DAST-C incorporated 
the attributes (or characteristics of scientists) identified by 
the foregoing researchers, and placed those elements in the 
upper portion of the checklist. These elements included the 
presence of lab coats, eyeglasses or other eyewear, facial 
hair, symbols of research, symbols of knowledge, and 
technology. Then they expanded the checklist items to 
include other attributes that seemed to frequently appear in 
drawings and, to some extent, reflected other things 
Chambers had noted in his work. Those additional checklist 
items included such as the ethnicity, gender, and age of the 
scientist, as well as the scientist’s environment (doing work 
indoors, for example, or in areas of secrecy and/or danger). 
The additional elements also included mythic stereotypes 


2016 
% of Images that #of #ofImages % of Images that 
are Scientists Images _ of Scientists are Scientists 
18.34 4,764 199 4.18 
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Table 3 
Scientists’ Disciplines in Book Images 


Year Discipline 

Earth Biology Life Physical 
2014 2 ] es 5 
2015 6 + 11 3 
2016 4 ps Ma 3 
Total 12 u 2 9 
(e.g., wild-haired and buzzing-eyed mad _§scientist- 


typologies or Frankenstein characteristics) and the presence 
of ideas represented by flashing light bulbs. These 
additional elements were included in the lower portion of 
the DAST-C. In a way, the lower portion of the DAST-C 
can help illuminate the human endeavor aspect of what the 
scientist is doing since it helps one derive information about 
the environment in which the scientist is working and 
something about what the scientist is doing. 

When using the DAST-C, the presence of any of the 
elements is scored with a “1” (even if there are multiple 
appearances of that same element in the image), and the 
total is then added to derive a score for the drawing. Scores 
can be determined for the upper portion of the DAST-C 
(the classic portion), for the lower portion, and for the 
overall total. Essentially, the higher the total DAST-C 
values, the more stereotypical the image is (although this 
does not necessarily or always translate into negative 
stereotypes, as might occur when one should see a scientist 
wearing a lab coat when working with chemicals). 

The DAST-C developers reported the instrument had a 
high inter-rater reliability (0.97-0.98) as determined using 
ANOVA procedures (Finson, Beaver, & Cramond, 1995). 
They also established instrument validity by conducting 
face-to-face structured interviews with subjects regarding 
their perceptions of scientists, and then compared interview 
results to those obtained just using the DAST-C. The 
developers’ conclusion was that the interview data closely 
matched DAST-C results, thus providing clear validity for 
the instrument. The instrument was later validated via other 
studies examining diverse populations such as racial groups 
(Finson, 2003), writing proficiencies (Monhardt, 2003), and 
national and cultural origin (Narayan, Park, Peker, & Suh, 
2013; Turkman, 2008) including studies conducted in not 
only the United States, but also in China, India, South 
Korea, Taiwan, Turkey, and the UK. 


Methodology 
The initial pool of picture books included for 
consideration in the study were those recommended by the 
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National Science Teachers Association in its NSTA 
Recommends book lists for the years 2014, 2015, and 2016. 
In total, 148 books were identified for examination. Not all 
books on the NSTA Recommends lists were considered 
picture books because the lists included chapter books for 
grades such as high school, and the focus of this study was 
on books for primary and middle school levels. (Note: 
Picture books are generally used in grades 3-5 and lower.) 
All 148 books were obtained through library sources. Since 
the researchers wanted to assess the appearance of scientists 
in the books, only those books that included images of 
scientists were selected for further examination. A total of 
56 books (37.83% of the 148 books) included images of 
scientists. In that subset of books, a total of 15,778 images 
were available for assessment, but only 1,676 (10.66% of 
the images) were actually of scientists. To be counted as a 
“scientist illustration,” the illustration had to either be 
showing a person traditionally perceived as a scientist or a 
person doing science of some sort. 

Once the pool of books to be studied was established, the 
number of illustrations within each book were counted. In 
total, 15,778 pictures were available for analysis. The 
books with the highest number of illustrations were ones 
that were a type of resource or glossary like a DK 
Publishing book and typically had many small pictures 
arranged in a collage-type of style on the pages. Any of the 
books containing at least one picture of a scientist was 
considered as available for analysis. 

Considering the large volume of pictures now available 
for analysis, a protocol was implemented to allow for easier 
management of picture assessment. For books having five 
or fewer scientist pictures, all pictures were analyzed. For 
books having more than five scientist pictures, five pictures 
were randomly selected from the book for analysis with the 
assumption that those five would not vary significantly in 
their characteristics from the other pictures in the book. The 
decision to randomly select scientist images for assessment 
from books containing many images was based upon 
sampling protocols often used in research studies. In our 
initial review of the books assessed in the study, we found 
the books having many images of scientists typically 
repeated images having the same scientist attributes without 
adding notable other attributes that would add to the 
measure of stereotypical elements. Our consensus was that 
assessing a specified number of randomly selected images 
from those books would yield the same whole-book DAST- 
C value as would be determined if all images in the book 
were included in the assessment. 

As an example, in the book Primates by Ottaviani and 
Wicks (2013), there were 495 pictures of scientists, 
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Table 4 
Presence of DAST-C Elements in NSTA Recommends Books 





Element 2014 

f % of Time DAST-C 

Element Appears in 
Year of Books 

Lab Coat 0 0.00 
Eyeglasses 17 19.54 
Facial Hair 18 20.69 
Symbols of Research Z 2.30 
Symbols of Knowledge 0 0.00 
Technology 0 0.00 
Male Gender 56 64.37 
Caucasian 83 95.40 
Indications of Danger 0 0.00 
Presence of Light Bulbs 0 0.00 
Mythic Stereotypes 1 KIS 
Indications of Secrecy 0 0.00 
Scientist Doing Work Indoors 6 6.90 
Middle-Aged or Elderly Scientist 37 42.53 
Total n = 87 


However, the entire book was about the work of Jane 
Goodall, Dian Fossey, and Biruté Galdikas, and the 
pictures essentially repeated the same set of characteristics 
throughout the book. Another protocol applied in the study 
for management purposes was to establish inter-rater 
reliability among the researchers on their assessments of the 
books. A random sample of 31 pictures was taken from the 
three years of books and scored independently by each of 
the three researchers using the instrument (DAST-C) 
chosen for analysis. Each researcher scored all 31 pictures, 
and those scores were then compared to determine inter- 
rater reliability, which was found via ANOVA procedures 
to be r=.84. Once this was established, the group was 
confident that images scored by any one of the researchers 
would be scored nearly the same as if scored by others in 
the research group. 


Table 5 
Basic Statistics for DAST-C on Stereotypical Elements in NSTA Recommends 
Books 








2014 2015* 2016* 
Upper DAST-C Mean 0.87 0.76 1.01 
Upper Range 0—1.80 0-1.67 0—3.00 
Upper SD 0.582 0.621 0.751 
Lower DAST-C Mean Geil pale 1.89 
Lower Range 1.0-3.6 1.2-3.4 1.0-3.0 
Lower SD 0.778 0.611 0.476 
Total DAST-C Mean 2.56 DSS 2ia7 
Total Range 04.8 0-4.4 0-4.00 
Total SD Voz 0.972 0.951 


*n of Scientist Images: 2014 = 677; 2015 = 800; 2016 = 199. 


School Science and Mathematics 


2015 2016 


% of Time DAST-C 
Element Appears in 


in % of Time DAST-C e 
Element Appears in 


Year of Books Year of Books 

10 9.43 6 8.11 

18 16.98 8 10.81 

20 18.87 19 25.68 

0 0.00 0 0.00 

0 0.00 0 0.00 

0 0.00 0 0.00 

78 73.58 53 71.62 

103 97.17 66 89.19 

0 0.00 0 0.00 

0 0.00 0 0.00 

0 0.00 0 0.00 

0 0.00 0 0.00 

20 20.75 if 9.46 

34 32.08 9 12.16 
106 74 


The DAST-C was then applied to each of the selected 
pictures. DAST-C scores were calculated for each picture. 
The scores of all the pictures in a single book were then 
used to calculate a mean DAST-C value for the book. This 
procedure also enabled the researchers to determine a 
DAST-C mean for all books within a given year of the 
NSTA Recommends lists. 


Analysis and Results 

The DAST-C was applied to each of the scientist images 
from the NSTA Recommends books. The data from this 
application are shown in Table 4, and some basic statistics 
are presented in Table 5. No statistical differences were 
found via ANOVA procedures for any element between the 
years. One interpretation of such a result is that the kinds of 
images included in the books selected by NSTA for its 
NSTA Recommends list remained relatively constant across 
the years. However, there are three particular elements in 
the DAST-C results (all from the lower portion of the 
instrument) that warrant further thought. The three focus on 
gender, race/ethnicity, and age. 


Conclusions/Discussion 

Stereotypical versus Nonstereotypical Portrayals 

Research about the presence and influence of 
stereotypical images of science and scientists is plentiful, 
and most often reveals stereotypical images are seemingly 
ever-present and very resistant to change (Finson, 2002). 
As a group, the illustrations of scientists in the NSTA 
Recommends lists would not be considered as being very 
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stereotypical based upon scores derived using the DAST-C. 
The data in Table 4 show that the means for each year 
(2014, 2015, and 2016) were below 3.0, which is a 
relatively low DAST-C score. This is a desirable result 
given science educators’ attention to reducing students’ 
stereotypical perceptions of scientists over the past 25 
years. Hence, one might conclude that the books in the 
NSTA lists were clearly overcoming the resistance to 
change noted in 2002 by Finson. Even so, the percentage of 
time facial hair and scientists doing their work indoors 
remained high compared to most other elements in the 
DAST-C. Conversely, it seems odd having an absence or 
near-absence of DAST-C elements on technology, symbols 
of knowledge, and symbols of research. Seeing an absence 
of elements alluding to symbols of danger or secrecy and of 
mythic stereotypes is a positive result. Such results might 
be attributed to the book illustrations showing scientists 
more often in outdoor settings where such stereotypical 
elements are less likely to exist. Given these points, there 
are three that bring one to pause. Stereotypical elements of 
gender, race/ethnicity, and age were clearly present in the 
books across all three years. 
Gender Issues 

The images of scientists portrayed in the MSTA 
Recommends books are predominantly male. Just over 64% 
of the images in 2014 were of males, and that percentage 
increased for 2015 and 2016 (both exceeding 70% of the 
images). Some might construe facial hair as a way to 
“double count” maleness, or as representative of style but in 
the case of the DAST-C the idea of facial hair as being 
stereotypical originally referred to the “mad” scientist type 
of image children were producing in their drawings. These 
results seem to run counter to efforts of science educators to 
be more inclusive with respect to gender representation. As 
far back as 1989, Fort and Varney noted that even though 
60% of the 1,600 students in their study were female, only 
about 10% of the drawings they obtained were of female 
scientists. Hence, little seems to have changed from that 
time up until 2016 with respect to female images of 
scientists in the collective consciousness. Notably, it is 
historically prevalent that females are very underrepresented 
in science fields and in science publications, and numerous 
efforts have been made to increase the role of females in 
science, beginning at the elementary school level and 
continuing post-college. One of the critical interventions for 
this has been providing role models and images of role 
models that are female. Where and when _ those 
interventions have been implemented, notable successes 
have been reported in changing students’ perceptions about 
who can be a scientist (Finson, Beaver, & Cramond, 1995). 
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In 2012, Farland-Smith examined the critical aspects for 
science identity construction in young girls dependent upon 
personal and social interactions between the girls and 
scientists. Bohrmann and Akerson (2001) similarly found 
that interventions including giving specific praise, utilizing 
guest speakers, viewing videos of scientists from minority 
populations, and ensuring equal participation in science 
activities by students of both genders resulted in students 
having fewer stereotypical images of scientists. 

Textbooks, trade books, and media have clearly been 
identified as significant influences on students’ formative 
perceptions about who scientists are and who can be a 
scientist (Farland-Smith, Finson, Boone, & Yale, 2014; 
Steinke et al., 2007). Picture trade books should include 
images of both males and females inasmuch it is 
historically appropriate. In addition, the images of persons 
included within a trade or picture book should be as non- 
stereotypical as possible (Farland, 2006a, 2006b). In their 
study, Farland-Smith et al. (2014) investigated the role of 
books (among other influences) in students’ formation of 
perceptions of scientists. They found that females in their 
study sample exhibited a higher level of sophistication in 
their perceptions of scientists—i.e., more expansive—as 
compared to the perceptions of male students. They also 
found that books and media both appear to be greater 
influences on males than females, whereas school is a 
greater influence on females than on males. This is an 
important point because trade and picture books children 
read contribute to their perceptions of scientists and their 
science identities. 

Race and Ethnicity Issues 

Perhaps more concerning is what one finds when looking 
at the race/ethnicity data from the NSTA Recommends 
books. Caucasian scientists by far outnumbered any other 
races/ethnicities in each of the years of books. In 2014, 
fully 95.4% of the images were of Caucasian scientists. In 
2015, it increased to 97.17%, and then slightly declined in 
2016 to 89.19%. As for gender issues, science in the United 
States has largely been perceived as an enterprise for 
Caucasians. Students from underrepresented populations 
have difficulty seeing themselves in the role of scientist, 
and are critically in need of images and role models that 
might support their perception that scientists can be from all 
races/ethnicities. 

Race and ethnicity have been issues with respect to 
perceptions of scientists for a very long time. In his 1983 
study, Chambers found that Caucasian students typically 
drew images of scientists who were also Caucasian, but 
minority students rarely drew images of minority scientists, 
and instead typically drew images of Caucasian scientists. 
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Sumrall (1995) found similar results. In her study 
comparing U.S. and Chinese student drawings, Farland- 
Smith (2009) found that a number of stereotypical 
misconceptions students possessed depended significantly 
upon the culture in which children learn. This was echoed 
earlier by Cavanaugh and Cavanaugh (2005) in their work, 
in which they concluded that for many children, what is 
presented to them is often interpreted in ways that help 
them define what they perceive to be culturally acceptable 
thinking and behavior (Cavanaugh & Cavanaugh) and have 
some effect of children’s idea of the word “scientist.” 
Clearly, if minority students cannot self-identify and 
connect with the illustrations of scientists in picture books, 
then there is a decreasing likelihood those students will 
develop positive science identities, take further science, or 
eventually seek science careers. 
Age Issues 

Finally, with respect to the age of scientists, the images in 
the books seem to be getting younger. In 2014, 42.53% of 
the images showed middle-aged or older scientists, declining 
to 32.08% in 2015 and dropping to 12.16% in 2016. The 
stereotypical and historic image of scientist has been of men 
of middle age and older. To many young learners, such 
perceptual images deliver the message that science is not for 
younger people. As with gender and race/ethnicity issues, 
serious interventions have been undertaken to dispel this 
perception. The trend in the NSTA Recommends books from 
2014 through 2016 show a very positive move in presenting 
scientists as increasingly younger people with whom 
students can more easily relate and—as a consequence—see 
themselves emulating in the future. 


Implications 

It is important that teachers plan student-centered 
activities that give them the opportunity to explore their 
world and encourage them to think of themselves as 
scientists. By explicitly selecting books reinforcing 
nonstereotypical characteristics of scientists, teachers can 
plan and deliver instruction that enables students to 
progress toward that goal and successfully address key 
components of state and national reading and science 
standards. It is similarly important that agencies such as 
NSTA take additional steps to ensure the books in their 
recommended reading lists reinforce nonstereotypical 
images of scientists inasmuch as possible, and go the 
additional step of including and promoting images that are 
more inclusive in terms of gender and race/ethnicity. 

It is likely many teachers rely on the guidance from 
respected book listings such as NSTA Recommends to select 
trade and picture books to use in their classrooms in their 


School Science and Mathematics 


efforts to convey to their students the idea that science is 
something accessible, and is a “human endeavor.” With 
regard to teachers and science educators choosing or 
recommending books from lists such as MSTA 
Recommends, it may be warranted they give additional 
attention to the way scientists are portrayed in the books 
rather than automatically accepting the books as appropriate 
due to their inclusion on the list. 

The results of this study indicate that science educators, 
teachers, professional organizations and publishers all need 
to be clearly aware of what constitutes stereotypical 
representations of scientists and work to minimize such 
representations in the materials they use with students. 
Employing such diligence will positively contribute to the 
ongoing efforts from the past 25 years to present science as 
a more inclusive discipline that is open to females and 
minorities. Such action will move science education 
forward, help ensure that national standards are met, and 
bring strengthening diversity to the field. 
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While conceptual understanding of properties, operations, and the base-ten number system is certainly associated with 
the ability to access math facts fluently, the role of math fact memorization to promote conceptual understanding remains 
contested. In order to gain insight into this question, this study looks at the results when one of three elementary schools in 
a school district implements mandatory automaticity drills for 10 minutes each day while the remaining two elementary 
schools, with the same curriculum and very similar demographics, do not. This study looks at (a) the impact that 
schoolwide implementation of automaticity drills has on schoolwide computational math skills as measured by the ITBS 
and (b) the relationship between automaticity and conceptual understanding as measured by statewide standardized 
testing. The results suggest that while there may be an association between automaticity and higher performance on 
standardized tests, caution should be taken before assuming there are benefits to promoting automaticity drills. These 
results are consistent with those that support a process-driven approach to automaticity based on familiarity with 
properties and strategies associated with the base-ten number system; they are not consistent with those that support an 


answer-driven approach to automaticity based on memorization of answers. 


Introduction 

The value of students being able to access math facts in a 
timely manner is well established (Caron, 2007; 
Woodward, 2006); however, approaches differ on how this 
should be attained. The National Council of Teachers of 
Mathematics (NCTM, 2014) suggests that the Common 
Core State Standards in Mathematics (CCSS-M) are 
designed so students can obtain access to math facts 
through procedural fluency obtained by an “understanding 
of properties, operations, and the base-ten number system” 
(CCSSI, 2010; NCTM, p. 73). This perspective would 
encourage students not to memorize .a math fact such as 
8+7=15 but rather to revisit strategies such as 
8+7=84+(2+5)=(8+2)+5=10+5=15 until the 
associated properties and operations become second nature 
to the student and the math fact is retrieved in a timely 
manner. 

Others believe that memorization of math facts is a viable 
approach to success in mathematics. Renaissance Learning, 
a company that has marketed materials, including 
MathFacts in a Flash, a system that includes computer- 
based math fact drills that could be used for memorizing 
math facts, asserts that: 
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Declarative memory (which recalls that things are so) 
not only speeds up the basic arithmetic operations 
themselves (Garnett & Fleischner, 1983), it also acts to 
“free up working memory capacity that then becomes 
available to address more difficult mathematical tasks” 
(Pegg, Graham, & Bellert, 2005, p. 50). (2015, p. 5). 


This study focuses on a school district where both the 
NCTM and the Renaissance Learning perspectives for math 
fact acquisition were present. The three elementary schools 
in the district are blue ribbon schools and administrators 
within the district have long valued the acquisition of math 
reasoning skills and selected Everyday Mathematics (The 
University of Chicago School Mathematics Project, 2007) 
as the district math curriculum in order to focus on math 
problem-solving skills over rote memorization of 
procedures (Fuson, Carrol, & Drueck, 2000). Teachers at 
these three elementary schools attended the same district- 
based professional leaming to assure fidelity with the 
curriculum and commonly engaged in practices to 
encourage sense-making including number talks, use of 
manipulatives, and individual/small group mathematics 
projects to explore mathematics connections and concepts. 
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Table 1 
Percent of Students Attaining Automaticity by School and Grade following 
the Automaticity Initiative at School 3 in the 2012-2013 Academic Year 





Grade School 1 School 2 School 3 
Third 23.88 20.69 80.00 
Fourth 66.22 45.65 82.76 
Fifth 60.00 HNP 83.33 


Over the past 15 years, it was quite common for this 
district to observe average percentile rankings in the 
math applications sections of the Iowa Test of Basic 
Skills (TBS) that were over 20% points higher than 
the average percentile ranking in the section on math 
computation. In response to this long-standing result, 
automaticity resources that could potentially encourage 
memorization were made available to elementary 
schools in the district with the expectation that this 
would improve results on the mathematics computation 
section of the ITBS. 

While all three elementary schools in the district had 
access to automaticity materials, only one school (School 3 
in Table 1) mandated 10 extra minutes per day of 
automaticity drills using math fact retrieval tools consistent 
with recommendations of the Institute for Educational 
Sciences (delineated in the following section). This was in 
addition to the normal classroom instruction using 
Everyday Mathematics. The other two elementary schools 


Table 2 
Measures Used in the Study 


Assessment 


Automaticity Tests 


continued concept-based instruction using Everyday 
Mathematics in their classroom instruction but, while they 
certainly provided “normal” encouragement for students to 
learn their math facts, did not mandate that classroom time 
be devoted to automaticity drills. While most automaticity 
activities consisted of individual students performing basic 
drills on a computer, teachers indicated that oral games, 
timed tests, songs, and mnemonic devices were also used. 
Table 1 presents the automaticity rates achieved in the 
study’s three schools by the completion of the 2012-2013 
school year as measured by the automaticity test described 
in Table 2. 

Automaticity rates at School 3 were over 80% for the 
third, fourth and fifth grades. As this was the only school to 
systemically incorporate automaticity drills into their 
curriculum, it is not surprising that their automaticity rates 
were the highest in the district, particularly in the third grade. 

These results provided an opportunity to study the impact 
of automaticity in two populations with similar 
demographics that used the same curriculum; one where 
automaticity drills were mandated and automaticity was 
achieved quickly, and the other where automaticity drills 
were not mandated and automaticity appeared to develop 
more slowly. Specifically, this study will examine 
individual and schoolwide data associated with these two 
within-district populations to ascertain the impact of 
mandated schoolwide automaticity. At the school level, we 


Description 


Automaticity with multiplication facts was determined by administering a three-minute test of 60 


single-digit multiplication items. A copy of a sample multiplication fact test is included in 
Appendix. Each student was given three minutes to complete as many problems as possible. They 
were graded to determine how many items were answered correctly in three minutes. If a student 
answered less than 55 items correctly in three minutes, the student was considered not automatic. If 
a student was able to answer 55 items correctly in three minutes, the student was considered 
automatic. If all 60 questions were answered correctly in three minutes, the student was considered 


perfectly automatic. 
Iowa Test of Basic Skills 
(ITBS—Mathematics 
Computation Section 
on the test level.” 
Kentucky Performance 
Rating for Educational 
Progress (KPREP) 


This section of the lowa Test of Basic Skills indicates that (ITBS, 2015): “each problem requires one 
arithmetic operation - addition, subtraction, multiplication, or division. The problems require 
operations with whole numbers, fractions, decimals, and various combinations of these, depending 


The most common assessment associated with understanding of mathematics concepts in Kentucky is 
the mathematics section of the KPREP which is described by the Kentucky Department of 
Education (KDE, 2015) as “a blended model built with norm-referenced test (NRT) and criterion- 


referenced test (CRT) items which consist of multiple-choice, extended-response and short answer 
items. The NRT is a purchased test with national norms and the CRT portion is customized for 
Kentucky.” Students are classified as novice, apprentice, proficient or distinguished based on the 
KPREP exam. In Kentucky performance reviews, scores of proficient and distinguished are 
considered consistent with an understanding of mathematics concepts appropriate for the students’ 
grade level. When describing students, the term proficient refers to students that are proficient or 
distinguished. The term not proficient refers to students that are classified as novice or apprentice. 





260 


Volume 117 (6) 


Automaticity and Conceptual Understanding 


will look at the impact of automaticity drills on schoolwide 
performance on the computation section of the ITBS. At the 
Student level, we will explore the broader relationship 
between automaticity and conceptual understanding of 
mathematics when a statewide standardized test is used to 
assess students’ conceptual understanding.’ Finally, we 
discuss some of the potential considerations associated with 
these results. 


Background 

Schoenfeld (2004) described mathematical competence 
as having both the knowledge of facts and “the ability to 
solve problems in new contexts or to solve problems that 
are different from the ones one has been trained to solve” 
(p. 262). How to help students in gaining this mathematical 
competence has been subject to many debates on what 
should be taught and how. Some of the catch-phrases 
associated with this debate include Math Wars, Back to 
Basics, and New Math. To be successful, students need 
procedural fluency that is “the ability to execute action 
sequences to solve problems” and a _ conceptual 
understanding, which can be considered “implicit or 
explicit understanding of the principles that govern a 
domain and of the interrelations between units of 
knowledge” (Rittle-Johnson, Siegler, & Alibali, 2001, 
pp. 346-347). Rittle-Johnson et al. (2001) in their study of 
fifth- and sixth-grade students found that procedural and 
conceptual knowledge can operate in concert among 
successful students. Many aspects of these discussions are 
mirrored in studies of how students should best learn basic 
arithmetic facts. 

The definition of automaticity as the “single-step retrieval 
of solutions from memory rather than some algorithmic 
computation” (Logan & Klapp, 1991, p. 179) is often 
associated with almost instantaneous retrieval of math facts. 
However, based on research by Isaacs and Carroll (1999), 
The MathFacts in a Flash materials used by the schools in 
this study recommended three seconds per problem for 
students in elementary school. This automaticity goal was 
adopted by the school district in this study and is important 
because it allows enough time per problem for students to 
use non-memory-based approaches when accessing math 
facts. 

The Everyday Mathematics curriculum, used by all 
three schools in this study, moves “from nearly exclusive 
emphasis on naked number calculation to developing 
conceptual understanding and problem-solving skills in 
arithmetic” (Everyday Mathematics, 2017a). Keeping 
with this goal, lessons in the Everyday Mathematics 
curriculum associated with multiplication facts use 
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patterns and structure to access math facts (Everyday 
Mathematics, 2017b). While Everyday Mathematics 
certainly promotes conceptual approaches to accessing 
multiplication facts, it also suggests the use of “written 
and choral fact drills’ (Everyday Mathematics, 2017c) to 
help students develop ready access math facts. Hence, 
multiplication fact drills are not necessarily inconsistent 
with Everyday Mathematics. However, as Everyday 
Mathematics is quite specific on what methods should be 
used to access math facts and as these specific methods 
do not include memorization, we would consider 
memorization of math facts to be inconsistent with this 
curriculum. 

Proponents of memorizing math facts often refer to the 
practice of overlearning which can be thought of as 
practicing that which has already been learned over and 
over again until it becomes second nature (Renaissance 
Learning, 2015). While overlearning can be used with 
conceptual strategies for obtaining math facts, proponents 
of memorizing math facts assert that a one-step, faster is 
better approach has benefits that multi-step cognitive 
strategies do not provide. With ready access to basic math 
facts, students are less likely to be overwhelmed by the 
amount of cognitive engagement problems often require 
and will be better positioned to solve more complex 
problems (Sweller, 1994). Several other studies (Ashcraft, 
1992; Campbell, 1987; Compton & Logan, 1991) assert 
that remembering memorized math facts quickly will free 
cognitive capacity and result in greater success with more 
complex problems. 

On the other hand, some studies discourage a 
memorization approach to math facts. Clark and Kamii 
(1996) found that multiplicative thinking appeared early 
and developed slowly in children and discouraged a 
“faster is better’ mentality toward math facts. An 
evaluation of student performance on the 2012 
Programme for International Student Assessment (PISA) 
found that the lowest performing students were those 
who used a memorization strategy (Boaler, 2015) and 
that success achieved through fact mastery is likely not 
achieved through memorization alone. More 
importantly, Marshall (2003) found that once facts are 
learned using rote memorization, it can be difficult to 
attempt to teach for understanding retroactively. 

Reviewing available research, a panel from the 
Institute of Educational Sciences (IES) (2009) for the 
“What Works Clearinghouse” acknowledges both 
answer-driven automaticity based on memorization and 
process-driven automaticity based on strategies and 
properties. They indicated it was important to attend to 
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conceptual underpinnings rather than “solely relying on 
rote memorization and drill and practice” (p. 43); noting 
that students should be able to use the distributive 
property to convert a more complex product into two 
simpler products. However, the panel also recommends 
spending “time after instruction with extensive practice 
on quick retrieval of facts through the use of materials 
such as flash cards or technology” (p. 43) for struggling 
students. The IES recommendation to extensively 
practice “quick retrieval of facts” with “materials such as 
flash cards or technology” was interpreted by the school 
district of this study as promoting “quick retrieval” with 
no restrictions on how to achieve it. In other words, 
while not necessarily to be encouraged, memorization of 
math facts was considered a viable approach to “quick 
retrieval.” Also, while the IES recommends potentially 
considering 10 minutes of automaticity drills for students 
that are struggling with basic math facts, one of the 
schools of this study widened implementation of 
automaticity drills to include all students. 


Research Questions 

We pose two research questions to provide insight into 
the ongoing debate on both the benefits of mandated 
automaticity drills and the benefits of automaticity itself. As 
the school district of this study was primarily interested in 
improving computation skills as measured by standardized 
tests, our first research question addresses whether 
mandated schoolwide automaticity drills are an effective 
strategy to achieve this. 

Research Question 1 (RQ1): What is the relationship 
between implementation of mandated automaticity drills for 
all students and schoolwide performance on mathematics 
computation standardized tests? 

As the school district had a history of success with 
conceptual understanding of mathematics as measured by 
standardized tests, we felt that this context for an 
automaticity initiative might provide new insight into the 
broader relationship between automaticity and conceptual 
understanding of mathematics topics. 

Research Question 2 (RQ2): What is the relationship 
between automaticity and conceptual understanding as 
measured by proficiency on statewide standardized tests? 

An important distinction to make between these two 
research questions is that the first seeks conclusions 
associated with the schoolwide implementation of 
automaticity drills and hence schoolwide results are the 
focus. The second looks for general associations between 
automaticity and proficiency and does not consider which 
school a student attends. 
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Study Design 
Study Participants and the Nature of the Automaticity 
Initiative 

All students in the third, fourth, and fifth grades during the 
2012-2013 academic year from the three elementary schools 
in this district participated in the study. Of these students, 
217 were third-grade, 230 were fourth-grade, and 224 were 
fifth-grade students. Roughly 93% of students in the district 
were Caucasian, 50% were female, 14% received free lunch, 
and 3% received reduced lunch. Over 99% of students who 
graduated from the one district high school that year went on 
to college, and the district was ranked in the 98th percentile 
in the state. The study participants attended three elementary 
schools that feed into the same middle school and are all 
within three miles of each other in the same demographically 
homogenous community. All teachers were certified and 
highly qualified according to Kentucky standards with no 
difference in recruiting practices, pay, benefits, training, or 
development opportunities for teachers at these three 
schools. All three schools are also characterized by very low 
staff turnover. 

In January 2013, the automaticity initiative described in 
the introduction was launched using the MathFacts in a 
Flash (which has now been renamed Accelerated Math 
Fluency) program with all third, fourth, and fifth grade 
students. A significant part of the initiative was computer 
drills. While all arithmetic facts were included in the overall 
initiative, multiplication math facts are the area of focus in 
this study. MathFacts in a Flash broke down multiplication 
facts into the following levels: 


Multiplication Multiplication Alternate Forms: 
by 0, 1 Review: 0to5 Multiplication by 6 to 10 
Multiplication Multiplication = Multiplication 
BY2295 by 6, 7 Review: 6 to 10 
Multiplication Multiplication = Multiplication 
by 4,5 by 8,9 Review 1 
Alternate Forms: Multiplication Multiplication 
Multiplication by 10 Review 2 
by 0 to 5 





Each level included the following: 

1. Baseline Test: Students completed a timed test for 
each new math level. 

2. Personalized Practice: Students practiced a variety of 
problems at their current level, including any problems 
missed on the previous test. 

3. Timed Tests for Mastery: Students completed 60-item 
timed tests. 

4. Automatic Advancement: Students moved to the next 
level after meeting their individualized goal on the test. 
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5. Instant Information for Teachers: Detailed reports 
gave teachers an instant snapshot of student progress. 

While MathFacts in a Flash was available to all three 
schools, two of the schools continued providing the same 
conceptual instruction concerning multiplication facts and 
simply provided access for students to voluntarily 
supplement their normal activities with automaticity drills. 
Similarly, the third school continued to provide the same 
conceptual instruction but mandated that all of its students 
receive 10 additional minutes of direct math fact instruction 
daily. It also included math fact instruction as part of its 
school improvement plan and mandated that all students 
participate in an end of year math fact tournament. These 
initiatives were considered consistent with the IES 
recommendation of “10 minutes a day” and learning math 
facts “through the use of materials such as flash cards or 
technology” and MathFacts in a Flash had advertised its 
product as being “‘in line with the IES Practice Guide.” It is, 
however, certainly worth noting that, in contrast to 
mandating automaticity drills for all students, the IES 
recommendations were limited to students struggling with 
mathematics. 

The results in Table 1 indicate a significant difference 
between end-of-year automaticity rates achieved at School 3 
where automaticity drills were mandated and Schools 1 and 
2 where automaticity materials were available but voluntary. 
Hence, as mandated versus voluntary automaticity drills 
produced significant differences in automaticity rates in 
populations that were otherwise very similar, this study 
explores other differences and similarities among students in 
these populations. As it is difficult to monitor approaches 
that students may take to learning math facts, the two 
populations of this study, students in schools where 
automaticity drills were mandated and students in schools 
where automaticity drills were voluntary, reflects a common 
reality for many classrooms, schools, and districts. 
Methodology 

Timeline and description of assessments used in this 
study. Table 2 presents the assessment tools that were 
used in this study. Table 3 provides the timeline for 


administering these assessments and for administering 
the automaticity initiative. 

Methodology addressing Research Question 1. In 
order to assess the impact of mandated automaticity 
drills on mathematical computation skills as measured by 
the ITBS, the following data were tabulated and 
analyzed: 

* The average national percentage on the computation 
section of the ITBS for each grade and school on the ITBS 
assessment before the automaticity initiative (in October of 
2012) and following the automaticity initiative (in October 
of 2013). 

* Preintervention to postintervention growth was 
measured by calculating, for each school, the change in the 
average national percentage from 3rd-graders in 2012 to 
4th-graders in 2013 and from 4th-graders in 2012 to 5th- 
graders in 2013. As students generally advance a grade 
from 2012 to 2013, this better captures growth among the 
same students as they advance. 

Methodology addressing Research Question 2. The 
authors certainly do not claim that proficiency on the 
Kentucky Performance Rating for Educational progress 
(KPREP) standardized test is a definitive indication of 
conceptual understanding of mathematics. However, from 
the released sample of questions (KPREP, 2012), we 
believe that many of the questions, particularly extended 
response questions that require students to show their work 
and explain their thinking, or questions that require the 
coordination of multiple representations, likely require 
more than memorization (Stein & Smith, 1998). So we 
consider proficiency, as measured by KPREP, is likely 
indicative, at least, of conceptual understanding beyond 
memorization of facts and procedures. Hence, while 
KPREP may be an imperfect absolute measure of 
conceptual understanding, we accept that a proficient 
student likely understands math concepts better than a non- 
proficient student so KPREP is effective as a relative 
measure in our study. In order to gain insight into the 
relationship between automaticity and conceptual 
understanding of mathematics as indicated by proficient or 





Table 3 

Timeline for Assessment Administration 

Date Assessment 

October 2012 Students took the preinitiative ITBS. 

January 2013—May 2013 Students participated in the automaticity initiative. 

May 2013 All students in all three schools took the final automaticity test that was used to assess 
automaticity levels in this study. 

May 2013 Students took the statewide standardized test to assess conceptual understanding (KPREP). 

October 2013 Students took the postinitiative ITBS. 
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Table 4 
Average National Percentile by Year and Grade of Students on the ITBS-Computation Assessment 

School 1 School 2 School 3 
Grade 2012-2013 2013-2014 2012-2013 2013-2014 2012-2013 2013-2014 
Third 66 71 66 62 D7. 53 
Fourth 62 61 58 50 JS) 50 
Fifth ah wes 62 71 59 56 
Average 68.3 68.3 62 61 ST. 53 
distinguished ratings on KPREP assessments, the KPREP Results 
and Automaticity Test results were used to perform the This section first presents results regarding the 


following descriptive statistics and analyses: 

* A table of the number of students, by grade, by 
automaticity level (not automatic, automatic but not perfect, 
or perfectly automatic) and by proficiency level (proficient/ 
distinguished on the KPREP exam or lower than proficient) 
was constructed and statistical analyses were performed 
for each grade to determine whether automaticity was 
independent of proficiency. If automaticity is independent 
of proficiency, being automatic would not be associated 
with being proficient. 

Where an association between automaticity and 
proficiency was found, follow up analyses explored the 
nature of this relationship. In particular, we explored to 
what degree proficiency might be expected to follow 
when students are automatic and to what degree 
automaticity might be expected to follow when students 
are proficient. To gain insight, quantitatively, into the 
degree to which being automatic implies being proficient, 
we tabulated: 

¢ The percent of automatic students that were proficient 
and 

¢ The percent of non-proficient students that were not 
automatic. 

Logically, automatic implies proficient (direct form) and 
not proficient implies not automatic (contrapositive form) 
are equivalent; however, they represent different 
populations in the data. A high percentage in both these 
tabulations support the conjecture that proficiency follows 
as a likely consequence of being automatic. 

Similarly, to assess the degree to which being proficient 
implies that one is automatic, we tabulated: 

¢ the percent of proficient students that are automatic 
(direct form) and 

* the percent of non-automatic students that are not 
proficient (contrapositive form). 

Once again, a high percentage in both these tabulations 
support the conjecture that being automatic follows as a 
likely consequence of being proficient. 
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relationship between mandated automaticity drills and 
schoolwide performance on the computation section of 
the ITBS that are associated with Research Question 1. It 
then presents general results on the relationship between 
student automaticity and proficiency as measured by 
statewide testing that are associated with Research 
Question 2. As the first research question explores the 
impact of mandated schoolwide automaticity drills on 
schoolwide performance on the ITBS, only schoolwide 
results are presented in the first section. As the second 
research question explores general relationships between 
automaticity and proficiency, students from all schools 
are treated as a single population and no school-specific 
results are presented. 

Mandated Automaticity Drills and Computation Skills 

The motivation for our first research question was that 
the district in this study expected higher automaticity rates 
to improve performance on the Mathematics Computation 
section of the ITBS. Hence, we started by tabulating the 
average national percentiles on the ITBS-computation 
assessments before and after the automaticity initiative to 
address whether this expectation is justified (Table 4). 

Looking at the average national percentiles for each 
grade and school on the ITBS-Computation Section, we 
can see from Table 4 that based on the preintervention 
(2012-2013) average over three grades, the ranking of the 
schools is: School 3 < School 2 < School 1. Based on the 
average postintervention (2013-2014) national percentages 
over three grades for each school, the ranking is also: 
School 3 < School 2 < School 1. Hence, the district and all 
of the principals felt that the best measure to assess 
performance would be to observe relative growth from 
preintervention to postintervention scores as the same 
students advance a grade. 

Table 5 shows the growth in the average percentile on the 
computation section of the ITBS as students proceed from 
the administration of the ITBS immediately before the 
automaticity initiative in 2012-2013 to the administration 
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Table 5 

Growth in Average National Percentile on the ITBS-Computation Assessment 
as the Same Students Advance a Single Grade from Prior to the Automaticity 
Initiative to Following the Automaticity Initiative 


School 1 School2 School 3 
Third grade to fourth grade —5 Pelé =7 
Fourth grade to fifth grade +11 13 4 
Average eek see wa =| 





of the ITBS following the automaticity initiative in 2013— 
2014 as the students advance a grade (see methodology for 
RQ2 in Section 4.2). On average, School 1 was the highest 
performing school of the three in terms of student growth. 
School 3 significantly outperformed School 2 in terms of 
student growth from the third to the fourth grade and 
School 2 significantly outperformed School 3 in terms of 
student growth from the fourth to the fifth grade. When 
considering the average of each school’s two percentile 
growths (Table 5), the ranking of the schools is School 
3 < School 2 < School 1. 

Automaticity and Proficiency 

Our second research question addresses the relationship 
between students’ automaticity and conceptual 
understanding as determined by the KPREP assessment. 
Table 6 presents the number of students by grade, by 
proficiency status and by automaticity level during the 
2012-2013 academic year. 

For each grade, a Pearson chi-square test of independence 
was performed on the 3 by 2 subtable associated with the 
given grade. In grade 3, automaticity and proficiency are 
independent so there is no significant association between 
automaticity and proficiency. The proficiency rates is 
slightly higher among the automatic students; however, this 
difference is not statistically significant (p > .25). In fourth 
and fifth grades, however, automaticity and proficiency are 
not independent (p<.01) so there is a_ significant 
association between proficiency and automaticity. 

Pursuing the nature of the association between 
automaticity and proficiency in the fourth and fifth grades, 
Table 7 presents the populations associated with the direct 
and contrapositive forms for the conjectures that (a) 


automaticity implies proficiency and (b) proficiency implies 
automaticity. 

For both conjectures, more than 70% of the associated 
populations satisfy the direct form of the conjecture. 
However, in both cases, less than 60% of the associated 
population satisfy the contrapositive forms for the same 
conjectures where 50% would represent a total lack of 
evidence either for or counter to the conjecture. 


Discussion 

This study focuses on students from _ three 
demographically similar schools that were within a short 
distance of one another, where all students used the same 
mathematics curriculum dedicated to conceptual 
understanding of mathematics concepts. As one of these 
three schools mandated 10 minutes of automaticity drills 
each day for every student in the school and the other two 
schools had no mandated automaticity drills, this provided 
a rich population of students that, while similar in terms of 
demographics and curriculum, were potentially different in 
the manner by which they learned their math facts. 

Research Question 1 addresses whether there is evidence 
that mandated schoolwide drills to promote rapidly 
achieving automaticity with multiplication facts improve 
computation skills as measured by the ITBS. While the 
three elementary schools in the district were very similar in 
most ways, School 3 was different from the other two 
schools in terms of (a) having mandated automaticity drills 
and (b) having significantly more automatic students (see 
Table 1). Correspondingly, an effective approach to RQ], is 
to observe the results of the ITBS using students from 
School 3 as an experimental group and students from 
Schools 1 and 2 as the comparison group. However, as can 
be seen from Tables 4 and 5, School 3 is the lowest 
performing school based on both average scores and 
preintervention to postintervention growth. Hence, 
regarding RQI1, there was no evidence that mandated 
automaticity drills, even when accompanied by higher 
automaticity rates, positively impact scores on the 
computation section of the ITBS. 





Table 6 
Number of Students by Automaticity Levels, Proficiency Status, and Grade in the 2012-2013 Academic Year 
Grade 3 Grade 4 Grade 5 
Not Proficient Proficient Not Proficient Proficient Not Proficient Proficient 
Not Automatic 46 86 4] 42 DH 33 
Automatic (not perfect) ¥ 23 oe 47 11 49 
Perfectly Automatic 10 32 6 62 13 78 
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Table 7 
Percentages Associated With the Association Between Proficiency and Automaticity 

Grade 4 Grade 5 
The percent of the automatic that are proficient (direct) 80% 84% 
The percent of the non-proficient that are not automatic (contrapositive) 59% 53 ae 
The percent of the proficient that are automatic (direct) % 72% 79 % 
The percent of the non-automatic that are not proficient (contrapositive) 49% 45% 





Research Question 2 addresses the relationship between 
automaticity and proficiency as measured on the statewide 
standardized test, KPREP. Although the third grade 
curriculum is rich with content associated with 
multiplication facts, there were only moderate differences, 
far from statistically significant, in the proficiency rates 
among the non-automatic and automatic groups of students 
in the third grade. In the fourth and the fifth grades, 
however, the difference in the proficiency rates between 
automatic and non-automatic students was statistically 
significant (p < .01) (See Table 6). As automaticity is not 
associated with proficiency in the third grade but is 
associated in the fourth and fifth grades, this would indicate 
that potentially, the nature of automaticity may have been 
different among students in the third grade versus students 
in the fourth and fifth grade. In the third grade, about two 
thirds of the automatic students were from School 3 where 
automaticity drills were mandated while about one third 
were from Schools 1 and 2 where they were not. Hence the 
majority of automatic students in the third grade had spent 
10 minutes per day on mandatory automaticity drills. In the 
fourth and fifth grades, however, the majority of automatic 
students were from schools 1 and 2 and had not been 
required to spend 10 minutes per day on automaticity drills. 
So it is possible that the lack of association in the third 
grade may be a result of a population of automatic students 
dominated by those that were required to perform 
automaticity drills. And the positive association in the 
fourth and fifth grades may be a result of a population of 
automatic students dominated by those that were not 
required to perform automaticity drills. 

These results are consistent with the NCTM perspective 
that the CCSS-M is designed to develop procedural fluency 
based on conceptual understanding over time. This 
perspective might suggest that quickly achieved 
automaticity without explicit attention to understanding the 
underlying properties and operations, is not expected to 
impact or be associated with KPREP results which are 
presumably based on the CCSS-M. With respect to the 
second research question, a new finding of this study was a 
yearlong delay before an association was identified between 
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multiplication math facts and proficiency in mathematics. 
This result would support Clark and Kamii’s (1996) 
assertion that multiplicative thinking in children takes time 
and a “faster is better” approach to math facts may not be 
optimal. 

There was an association between automaticity and 
proficiency in the fourth and fifth grade so, as part of RQ2, 
we explored the nature of this association. Table 7 shows 
that over 80% of fourth and fifth grade automatic students 
were proficient. However, Table 7 also shows that on 
average 56.5% of non-proficient fourth- and fifth-graders 
were not automatic. As there were a significant number of 
non-proficient students in grades four and five and as 50% 
would represent a completely neutral finding, we felt that the 
data for the contrapositive form of the conjecture were too 
weak to support the conjecture that automaticity implies 
proficiency. Similarly, Table 7 shows that about three fourths 
of fourth- and fifth-grade proficient students were automatic; 
but, less than half of non-automatic fourth- and fifth-graders 
were not proficient. Therefore, the data actually slightly 
contradict the conjecture that proficient implies automatic. 
As there were a significant number of non-automatic 
students in grades four and five and the data associated with 
the direct and contrapositive forms of the conjecture were 
contradictory, we did not feel there was evidence to support 
the conjecture that proficiency implies automaticity. So the 
data in this study supported neither proficiency as a 
consequence of automaticity nor automaticity as a 
consequence of proficiency among fourth- and fifth-graders. 
The most we could conclude in this study regarding the 
association between proficiency and automaticity among 
fourth- and fifth-graders was that there is a tendency for 
students in this district to share both of these characteristics 
after they have advanced beyond the third grade. 

The results of this study can provide insight to teachers 
and administrators that are contemplating automaticity 
initiatives to improve mathematics standardized test scores. 
This study, as with others, did affirm an association 
between automaticity with multiplication facts and 
mathematics proficiency as measured by standardized tests. 
However, the results of this study would suggest that the 
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process by which automaticity is attained may be 
important. 

As we discussed in the Introduction and Background 
sections of our article, proponents of a “faster is better” 
approach to retrieving of math facts often suggest that 
replacing a more extended cognitive process (figuring out a 
math fact through properties and operations) with a simple 
memorized result may free up cognitive capacity that can 
then be used to more effectively solve complex problems. 
We refer to such automaticity, achieved through 
memorization, as answer-driven automaticity. Counter to 
proponents of answer-driven automaticity, the work of 
Rohrer, Dedrick, and Burgess (2014) found that better 
long-term results were obtained when students revisit topics 
multiple times through diverse problems and contexts. 
Rather than overlearning answers for sets of very similar 
problems, this approach promotes procedural fluency 
through broad familiarity of associated strategies and 
properties. Hence, rather than focusing on quick retrieval of 
answers, this approach would focus on quick retrieval of 
properties and operations with the development of flexible 
thinking processes to apply them. We refer to automaticity 
borne of familiarity with strategies and properties as 
process-driven automaticity. We would consider process- 
driven automaticity to be consistent with the NCTM 
assertion that the CCSS-M is designed to achieve 
procedural fluency through familiarity with the properties 
and operations of the base-ten number system. 

Our data indicate that automaticity that is more quickly 
obtained through automaticity drills does not appear to be 
associated with conceptual understanding as measured by 
standardized tests. Our data did indicate that automaticity 
obtained over a longer period of time, perhaps through 
familiarity with the properties and operations of numbers as 
the NCTM suggests should happen with the CCSS-M, is 
associated with conceptual understanding as measured by 
standardized tests. In the context of a broader discussion 
on process-driven automaticity versus answer-driven 
automaticity, our data would suggest that students would 
best benefit from the development of process-driven 
automaticity. Hence, when making decisions associated 
with automaticity interventions, administrators and teachers 
may wish to consider whether the intervention promotes 
process-driven or answer-driven automaticity. 


Future Directions 
This study has important implications for further research 
because it calls for more evidence to unfold the benefits and 
long term effects of process-driven automaticity 
approaches. While our study was quantitative in nature, we 
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performed related observations of an automatic proficient 
student and an automatic non-proficient student that 
received conceptual classroom instruction from the same 
mathematics teacher; however, both experienced 
supplemental mandated automaticity drills. When asked to 
make up a number story to show the meaning of 2 X 6, the 
proficient automatic student spoke of two plates with six 
cookies on each plate. This would represent the highest 
level cognitive construct (Clark & Kamii, 1996), 2 x 6 
represents two sixes. However, the automatic non- 
proficient student responded “I have six cards and my 
friend has six cards. What is that put together?” This is 
certainly correct, however, it still addresses multiplication 
as repeated addition which is a lower cognitive construct 
(Clark & Kamii, 1996). Particularly in cases where 
conceptual understanding is partnered with math fact drills, 
an intriguing future direction might be qualitative research 
on whether mandated automaticity drills may shift students 
from process-driven to answer-driven automaticity, 
potentially stunting their conceptual development. 

This study provides insight into the impact (or lack 
thereof) of automaticity drills among students that are, in 
general, not struggling with mathematics. However, the 
characteristics of the participating schools should be taken 
into account when generalizing results. All three schools in 
the study were Blue Ribbon schools with above average 
student performance and all three schools were invested in 
building a strong mathematical foundation for students based 
on conceptual understanding. In future studies, it would be 
interesting to see whether these results would be replicated in 
a lower performing school district committed to conceptual 
understanding or in school districts with more procedural 
approaches to mathematics education. The recommendations 
provided by IES mainly targets struggling students, so it may 
be interesting to,see how all the associations explored in this 
paper would look in such a setting. 
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Authors’ Note 


‘It should be noted that the authors do not necessarily 
endorse this standardized test as a single definitive measure 
of conceptual understanding; however, this was the 
available instrument designed for this purpose. 


Appendix 
Sample Multiplication Facts Automaticity Test 
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Afterschool programs have garnered much attention as promising environments for learning where children can engage 
in rich science activities. Yet, little is known about the kinds of instructional materials used in typical, large-scale 
afterschool programs that implement science with diverse populations of children. In this study, we investigated the types, 
features, and use of science instructional materials at more than 150 public afterschool program sites across California. 
Using afterschool site survey data, we categorized the types and the range of materials used at the sites. We then collected 
a subsample of the instructional materials for in-depth analysis of their support features for enabling staff and children to 
enact science. We also interviewed afterschool site staff to better understand how they selected and used materials. Results 
from our analysis of survey and interview data show that afterschool staff primarily used stand-alone lessons and activities 
found on the Internet or in activity books as resources for planning and enacting science. Our analysis of the subsample of 
instructional materials indicate that curricular materials, while used less frequently by afterschool staff, have on average 
more of the support features that would help them implement high-quality science experiences with children. 


AFTERSCHOOL programs are increasingly seen as 
important settings where ambitious science learning can 
occur. Within the past 15 years, funding for development 
and research on out-of-school programming for science has 
dramatically increased, resulting in the creation of new 
programs and instructional materials for afterschool science 
(e.g., Lawrence Hall of Science, 2009; Lyon, Jafri, & St. 
Louis, 2012) as well as a growing body of research 
literature (e.g., National Research Council [NRC], 2009). In 
the area of afterschool programming, research shows that 
afterschool settings can play a critical role in providing 
children with access to rich science learning opportunities 
by substantially expanding the time children have to 
explore science (Coalition for Science ‘After School, 2007). 
These opportunities can be considered especially important 
for elementary-aged children, given the education policy 
context wherein science instruction in elementary schools is 
often undervalued and its presence is modest (Banilower 
et al., 2013; Marx & Harris, 2006). 

Beyond simply extending the school day, or providing 
more school-like instruction after school, afterschool 
settings offer a unique learning environment often quite 
different from the traditional school structure (Eccles & 
Templeton, 2002). While afterschool programming has 
increasingly focused on academic enrichment, the activities 
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are often infused with a youth development perspective that 
emphasizes active and collaborative learning, developing a 
sense of mastery, and meaningful participation (Mahoney, 
Larson, & Eccles, 2005). By blending science education 
and youth development perspectives, many programs have 
demonstrated positive effects, such as improved attitudes 
toward science and science-related fields and careers, 
increased science knowledge and skills, and a higher 
likelihood of graduating and pursuing a science-related 
career (Chi, Snow, Goldstein, Lee, & Chung, 2010). 

In recent years, state educational agencies and funders 
have begun to invest in large-scale afterschool initiatives. 
As these initiatives are realized, it will be important to 
examine how they support science. At the present time, 
little is known about how to implement high-quality 
science offerings on a broad scale in typical afterschool 
programs that are not expressly designed as environments 
for science learning (Noam et al., 2010). There is, for 
instance, little research on the kinds of instructional 
materials used in typical, large-scale afterschool programs 
that provide services to diverse populations of children and 
implement science in their settings.' There is also a need for 
research on how the instructional materials that afterschool 
programs do use align with contemporary views on how 
learners can best be supported in out-of-school settings, 
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such as the principles for science learning described in the 
National Research Council’s synthesis report on informal 
science learning (NRC, 2009). 

In an effort to better understand the conditions under 
which science is provided in large-scale afterschool 
initiatives, we investigated the use of science instructional 
materials within and across sites of a statewide afterschool 
program in California. The publicly funded initiative 
provides support for community-based afterschool 
programming in all regions of the state. Our aim was to 
gain insight into the types of science instructional materials 
in use at the afterschool sites, the range of use of different 
types of materials, the kinds of support features provided 
within these materials, and how afterschool staff used the 
materials to plan and enact science. 


Background 
Afterschool Science 

The field of afterschool science is growing and garnering 
increased attention as educators, researchers, and 
policymakers recognize that the hours after school can 
provide unique time and structure for engaging children in 
science. In response, an impressive array of stakeholders— 
foundations, government agencies, informal science 
institutions, universities, and national youth organizations— 
have made investments in providing afterschool 
opportunities in science. Many recently developed initiatives 
serve diverse youth populations and aim to increase the 
participation of low-income youth and underrepresented 
groups in science, notably minority youth and young 
women (e.g., Lyon & Jafri, 2010). 

Contemporary views on how learners can best be 
supported in out-of-school settings have been highlighted in 
the National Research Council’s (NRC) synthesis report, 
Learning Science in Informal Environments (NRC, 2009). 
The report illustrates how informal learning environments can 
play a distinct and substantive role in promoting meaningful 
engagement in science. Moreover, the report recommends 
that informal learning environments have clear learning goals, 
are highly interactive, include multiple pathways for children 
to engage with science, facilitate science learning across 
settings, promote engagement and learning over extended 
time, and draw upon children’s interests, experiences, and 
prior knowledge to support rich learning (NRC, 2009). These 
recommendations reflect the emerging consensus on key 
principles that bring together views on science learning with 
youth development perspectives. 

Although there is generally wide agreement about the 
tremendous potential for afterschool programs to 
accommodate powerful science learning experiences, there 
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is evidence that these settings have their own challenges 
that can make it hard to enact science well. For instance, 
afterschool staff members responsible for enacting science 
with children often have neither a science background nor 
an understanding of how to orchestrate rich science 
activities (e.g., Noam, Dahlgren, Larson, & Dorph, 2008). 
Moreover, high turnover of staff is a persistent problem 
(Dennehy & Noam, 2005) that thwarts efforts to provide 
long-term professional development. Even when science- 
oriented professional development is offered, it tends to be 
limited in terms of time and material resources, and of 
variable quality (e.g., Noam et al., 2008). 

Science Instructional Materials 

The term instructional materials encompasses a wide 
range of resources that can be used by educators to plan for 
and enact learning experiences for students. In general, they 
are the concrete resources—activity descriptions, lesson 
plans, and curricular modules—that provide educators with 
activities and guidance to help students accomplish 
particular aims (Ball & Cohen, 1996). In science education, 
these materials can take many forms including 
commercially based activity books that provide collections 
of science experiments or activities, science project kits 
such as butterfly hatching kits or model rocket kits, media 
resources such as educational science videos, trade books 
such as children’s science books, and stand-alone lesson 
plans and activities downloaded from the Internet. They 
also can take the form of curriculum materials that organize 
activities and lessons into a coherent sequence of 
experiences that support learning over an extended period 
of time, usually over days or weeks, but sometimes over 
months and even years. While instructional materials can 
vary widely in design and purpose, an _ essential 
characteristic is that they provide some level of pedagogical 
support to educators (Ball & Cohen, 1996). 

An important part of science instruction is using materials 
purposefully to make science accessible for students (Harris 
& Rooks, 2010). Instructional materials have been widely 
acknowledged for their central role in supporting learning in 
school-based environments (Ball & Cohen 1996) and have 
had a role in out-of-school learning environments, too (Katz 
& McGinnis, 1999). Well-designed materials with embedded 
support features can help educators with creating particular 
kinds of learning experiences (Singer, Marx, Krajcik, & 
Clay-Chambers, 2000) and even provide opportunities for 
educators to learn themselves as they teach (Davis & Krajcik, 
2005). Support features can take the form of steps to follow, 
rationales for suggested actions, prompts or questions for use 
during instruction, and descriptions of outcomes to expect. 
Carefully designed support features have clear advantages— 
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they can provide background information on science content, 
give guidance on arranging the learning environment, 
Suggest strategies for interacting with students, and help 
educators anticipate student thinking. Of course, not all 
instructional materials are designed well. Some can be overly 
prescriptive, thereby diminishing the decision-making role of 
educators and the autonomy of students (Shulman, 1983). At 
the other end of the spectrum, underspecified materials can 
make it difficult for educators to envision how the leaming 
experience should unfold (Kauffman, Moore Johnson, 
Kardos, Liu, & Peske, 2002). 

Educators’ use of instructional materials has been a focus 
of research in school settings, and much of this research 
highlights that educators must thoughtfully interact with 
materials to make instructional decisions that meet the 
needs of their students. Following lesson or activity plans 
step-by-step may not lead to a successful learning 
experience for students (Davis & Varma, 2008). Likewise, 
straying too far from the lesson or activity structure risks 
that students will benefit less from the materials or not 
entirely meet the learning goals of the activity. Thus, in 
order to enact instructional materials effectively, it is 
important for educators to make interpretations and 
modifications consistent with the purposes and structures of 
the materials and consistent with their youths’ learning 
needs and interests. These types of decisions, along with 
deciding how and when to enact lessons, are part of the 
important planning process that all educators must go 
through (Ben-Peretz, 1990). To date, there has been little 
research conducted on the role of instructional materials in 
supporting ambitious afterschool science. 


Study Context 

The work reported here is part of a multi-year study 
designed to investigate the nature of afterschool science 
offerings in a statewide afterschool program system in 
California, including the sources of support for science 
programming and afterschool staff development at program 
sites. California’s After School Education and Safety 
(ASES) program supports “locally driven” afterschool 
services at schools and in other community settings. 
Afterschool sites within the system are organized at the 
community level and run by partnerships between schools 
and afterschool providers, typically youth and community- 
based organizations. More than 4,000 afterschool sites are 
funded through the program and serve over 400,000 youth 
in grades K-9 each year, Each ASES site serves children 
from a partner public school; only schools with 50% or 
more of their students eligible for free or reduced-price 
lunch can participate in the program. Most sites (89%) are 
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affiliated with elementary schools that serve students from 
Kindergarten through Sth or 6th grade. In the present study, 
we sought to better understand the types and range of 
science instructional materials in use at the afterschool sites 
serving elementary-aged children, the kinds of supports 
provided within these materials, and how afterschool staff 
used the materials to plan and enact science. Our research 
questions were: 

1. What are the types and range of the science 
instructional materials used by staff and participants across 
sites of a publicly funded statewide afterschool program? 

2. What are the support features of the science 
instructional materials used by staff and participants in the 
afterschool program? 

3. How do staff members select and use science 
instructional materials to plan and enact science activities at 
their afterschool program sites? 


Methods 

To address our research questions, we designed and 
administered a survey to a sample of afterschool sites 
serving elementary-aged children within the statewide 
ASES program. The afterschool science survey asked site 
coordinators to report on their site’s science offerings, 
including the science instructional materials used by staff 
and participants. Next, we categorized the range of 
instructional materials identified by sites in the survey data. 
We then selected and gathered a representative sample of 
instructional materials and conducted an analysis of support 
features. Finally, we conducted semi-structured interviews 
with a sample of afterschool site personnel on how they 
selected materials and how they used them to plan for 
afterschool science. 

Administering the Afterschool Science Survey 

The afterschool science survey contained questions on 
frequency of science offerings and typical activities, types 
of instructional materials used, supports for activities, 
partnerships with science providers, and contextual factors 
related to capacity to deliver science instruction, such as 
staff at the sites and their qualifications. It also included 
questions regarding how decisions are made about science 
programming at the site level, including the staff person(s) 
responsible for selecting materials, activities, and science 
topics, and what factors shape their decisions. 

Sample. To select our survey sample, we collected a 
stratified random sample of urban and rural ASES sites, 
with sites across the two strata matched on school-wide 
demographics and achievement variables. A_ stratified 
random sample of 600 sites was drawn from the roughly 
3,700 elementary sites that comprised 89% of all the 
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publicly funded afterschool sites in California’s ASES 
program at the time of the administration of the survey. 

Comparison of the survey sample to the total population of 
ASES elementary afterschool sites indicated that the two 
groups were equivalent in terms of proportion of students 
eligible for free or reduced-price lunch, average Academic 
Performance Index (API) scores, average parental education, 
proportion of their school’s teachers who were credentialed, 
and proportion of students in their school identified as gifted/ 
talented. It is important to note, however, that the ASES 
elementary school-age population itself differs from the 
population of California elementary schools in a number of 
ways. Because only schools with more than half of their 
students eligible for free or reduced-price lunch are eligible 
for ASES afterschool programming, ASES elementary sites 
tend to have more non-White students, lower API scores, and 
fewer students identified as gifted/talented compared to 
California elementary schools as a whole. 

Procedure. The survey was sent to 600 ASES site 
coordinators (i.e., on-site program supervisors) at each 
selected afterschool site during the 2010-2011 school year. 
The survey was administered both online and on paper, 
with the majority of respondents completing the survey 
online. The survey was in the field for a total of 15 weeks; 
426 site coordinators completed the survey (71% response 
rate) during this time period. After eliminating incomplete 
surveys and surveys with errors, the final sample was 415 
surveys. 

Categorizing the Types of Instructional Materials 

On the survey, site coordinators were asked whether or 
not they had offered science at their program site during the 
school year. Coordinators who indicated that they offered 
science were asked to name up to three different science 
instructional materials that were in use at their sites and that 
were representative of the kind of science resources 
typically used by staff; to briefly describe them; and to 
identify the source and publisher. We used this survey data 
to identify and categorize the types of instructional 
materials used by afterschool program sites. 

Selection. We filtered our set of survey responses to 
create a subsample of program sites that had listed science 
instructional materials. First, we filtered on whether and 
how often sites offered science. From our 415 survey 
responses, 380 afterschool sites reported offering science. 
Next, we filtered for whether the sites had offered science 
during the school year in which we administered the survey 
and 294 sites remained. Finally, we filtered for whether the 
remaining sites listed science materials, and found that 159 
sites nominated at least one material used in their after 
school science programming. This sample represented more 
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than half of the sites that offered science during the school 
year and provided a rich and varied dataset for analysis. 

Categories. The materials identified from the survey 
responses were each sorted into one of three broad 
categories: (a) science curricular materials, (b) science 
enrichment materials, or (c) non-materials. Curricular 
materials were defined as science units or modules 
comprised of sequenced activities with specifications for 
enactment over multiple sessions. These materials included 
teacher/instructor guides and provided opportunities for 
science learning over an extended period of time. 
Enrichment materials encompassed a broad array of stand- 
alone lessons and activities designed for short-term use. 
Instructional materials that were sorted into the enrichment 
category included prepackaged science projects, stand- 
alone activities or science lesson plan activities drawn from 
websites or books, and science trade books or media that 
were used to support science learning. These materials were 
typically not based on a logical learning sequence over time 
(e.g., related topics over multiple sessions that build upon 
each other) even though some activities within the books 
and websites were organized by science topic. Non- 
materials were defined as responses that were not science 
instructional materials or instructional activity resources for 
enacting science with children. These responses typically 
were science topics such as “weather” or “insects” that did 
not include any accompanying information regarding the 
instructional materials, or were externally delivered science 
events and fieldtrips. 

We further delineated material types within our two 
broad focal material categories. For Curricular materials, 
we sorted each into one of three subcategories: (a) Out-of- 
school curricula (C1), (b) School-based curricula (C2), and 
(c) Context-independent curricula (C3) where the material 
was developed for cross-setting use (i.e., either in-school or 
out-of-school contexts). Enrichment materials were sorted 
into one of five subcategories: (a) Science lesson or activity 
plans from websites (El), (b) Science lesson or activity 
plans from books (E2), (c) Trade book or media (E3), (d) 
Prepackaged science project (E4), and (e) Site-developed 
material or activity (E5). Materials identified from the 
survey responses that did not have sufficient information to 
assign them to a materials category were designated as “not 
codable for material type.” 

Procedure. The 159 sites nominated a total of 275 
science instructional materials. All the nominated materials 
were identified and collected as either a digital or physical 
copy. As a first step in the analysis, the nominated materials 
were independently coded into the eight categories listed 
above by two researchers, one who categorized the full set 
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of materials and another who categorized a randomly 
selected subset of the materials (20%, n = 55). Afterward, 
the two researchers met to compare their categorizations, 
check for reliability, and calibrate their decision rules. When 
both were uncertain about a categorization, a third 
researcher provided judgment to help clarify and make a 
decision. As part of this process, the research team modified 
and refined the categories and their parameters. We then 
conducted a second round in which the two researchers 
revisited category assignments and either confirmed or 
reassigned materials based on the refined category 
parameters. The inter-rater agreement was 87%. Differences 
were resolved through discussion and consensus to reach a 
final categorization for each material. 
Coding the Support Features of the Materials 
Selection of materials. We used the results from our 
categorization of the instructional materials as the basis for 
selecting a subsample of materials for further analysis. We 
purposefully selected a proportional number of materials 
from each of five subcategories—the three subcategories of 
curricular materials (out-of-school materials, school-based 
materials, and context-independent materials) and two 
subcategories of enrichment materials (lessons/activities from 
websites [E1] and lessons/activities from books [E2])}—that 
were representative of the majority (73%) of the materials in 
the overall material sample. We also ensured that the types of 
materials selected for coding were representative of the 
overall pool of nominated materials. Of note is that other 
enrichment subcategories did not lend themselves to further 
sampling and coding (1.e., site-developed materials [ES] 
would not be generalizable and trade books [E3] and 
prepackaged kits [E4] did not occur frequently) and thus were 
dropped from subsequent analysis. In all, 37 instructional 
materials were selected for in-depth support feature coding. 
Coding scheme. We designed a comprehensive coding 
scheme to identify and analyze the full range of support 
features provided within the instructional materials. This 
coding scheme was based on the strands from the Learning 
Science in Informal Environments report (NRC, 2009) and 
Taking Science to School (NRC, 2007), as discussed in 
more detail below. The scheme measures instructional 
support across four dimensions: (a) content and structure of 
the activities, (b) usability for facilitators and children, (c) 
engagement for children and relevance to their lives, and 
(d) promotion of scientific thinking, reasoning, and 
practice. For each dimension, we specified a set of 
indicators for identifying support features within science 
instructional materials. A total of 61 codes represented the 
indicators for these four dimensions (see Appendix). 
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Dimension 1, content and structure of the activities, 
examines the features that help facilitators to structure the 
science activities into a coherent “storyline” for children to 
move along a pathway for learning. Indicators for this 
dimension are 14 codes that focus on such features as 
science learning goals, sequence and pacing, and allocating 
time for doing science. Dimension 2, usability, addresses 
features in materials that support enactment and accessibility 
for facilitators and children. This dimension contains 23 
codes that address facilitator features, such as preparation for 
instruction, guidelines for orchestrating activities, and 
educative supports for effective enactment, as well as support 
features for children, such as directions and materials (e.g., 
activity sheets, science notebook, directions/procedures, and 
posters), and specialized supports for children of diverse 
backgrounds and abilities to access the science. 

For Dimension 3, engagement for children and relevance 
to their lives, we established indicators for identifying 
features that attempt to bridge science to children’s 
everyday social and physical world. These 11 codes include 
features for facilitators to situate learning in a context, such 
as a driving question, dramatic scenario, real-world 
situation, or compelling problem/issue; help children use 
their prior knowledge and interests for learning, and 
provide opportunities and support for children to make 
choices. Dimension 4, promotion of scientific thinking, 
reasoning, and practice, encompasses supports and prompts 
for facilitators to help children to think and reason about 
their science experiences and to effectively reflect on their 
learning. The 13 codes for this dimension include supports 
for facilitators and children to engage in science practices 
and productive discourse that encourages collective 
scientific thinking via discussion and debate. 

The dimensions were constructed from the National 
Research Council’s research synthesis reports on learning 
science in school (NRC, 2007) and out-of-school settings 
(NRC, 2009). Indicators were vetted against a review of 
research literature to provide an argument for the selection 
and inclusion of specific indicators under each dimension. 
It is important to note that the purpose of this coding 
scheme was not to determine the quality of instructional 
materials, but to identify important features of materials that 
could support the enactment of science. 

Procedure. We collected the full set of instructional 
materials that comprised our subsample for coding. We used 
a small subset of the materials for refining the coding scheme 
and training on its use. Two researchers were trained on 
using the coding scheme and went through multiple practice 
rounds of coding with this smaller set. Once sufficient 
agreement—above 70% on the majority of the codes 
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(Stemler, 2004)—was reached in the practice rounds, each 
researcher was randomly assigned approximately 60% of the 
materials in the main set, with 20% of the materials coded by 
both researchers for reliability checking. The researchers met 
to compare codes, reach consensus on any disagreements, 
and assign final codes for all indicators. When both 
researchers were uncertain about a piece of evidence for 
coding a support feature, a third researcher provided 
judgment to help clarify and reach agreement. 

We examined the 37 instructional materials at the lesson or 
activity level and conducted double sampling of lessons/ 
activities when a given material included more than one 
lesson in order to account for some amount of range within 
the material (such as a book of science experiments or a 
textbook). In total, we coded 67 lessons/activities: 17 website- 
based lessons/activities (E1), 32 book-based lessons/activities 
(E2), 8 out-of-school curriculum-based lessons/activities (C1), 
8 in-school curriculum-based lessons/activities (C2), and 2 
context-independent curriculum-based lessons/activities (C3). 
Interviewing Afterschool Site Staff 

Our final step to understanding the role of instructional 
materials in afterschool science was to examine how staff 
members selected materials and the materials’ role in 
planning for science. We conducted interviews with staff 
from 13 afterschool sites to better understand how these 
materials were situated within the science offerings 
provided at afterschool sites. 

Selection of sites. Our selection process began with a 
sample of 119 candidate sites whose survey responses 
indicated that they had two or more sources of support for 
science, offered science at least once a week, and used 
science instructional materials. We further narrowed down 
our list by identifying sites that provided the most frequent 
science offerings and collectively represented a wide range 
of materials. We then contacted 36 of the most promising 
sites, resulting in 13 sites with staff members who agreed to 
participate in interviews. 

Interview protocol. We developed a semi-structured 
interview protocol that addressed such topics as staff and 
site capacity for offering science, the process of selecting 
instructional materials, and how sites used materials to plan 
and prepare for science. To learn about staff capacity, the 
protocol included questions about site coordinators’ and 
facilitators’ experiences and perspectives on leading science 
activities with children, backgrounds in science, and 
traning for afterschool science. Questions about site 
capacity included frequency of science offerings, and the 
physical space, materials and other resources available for 
science. The protocol also prompted interviewees to 
describe a recent or upcoming science activity as an 
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entryway to conversation about materials selection and the 
activity planning process. Questions during this phase of 
the interview probed for the kinds of instructional materials 
typically used, the reasons and criteria for selecting them, 
constraints influencing the material selection process, the 
ways in which the materials were used to plan, and how 
they were used to enact science with children. 

Procedure. Interviews were conducted via phone with 
afterschool site coordinators or facilitators who had 
knowledge of the process of selecting materials and 
planning for science at their sites. The interviews, lasting 
45—60 minutes, were audio-recorded and then transcribed. 

We used an inductive approach to identify themes within 
and across and transcripts. Topic codes were assigned to 
statements made related to different topics in each set of 
interview data, then compared across interviews, and finally 
condensed into a summary format. Each topic summary 
included exemplars that were representative of the interview 
responses. This process was carried out by two researchers 
who independently assigned topic codes and then met to 
discuss, resolve differences, and construct the summaries. 


Results 
Types and Range of Science Instructional Materials 

Although 159 sites nominated materials in their survey 
responses, 13 sites were dropped from subsequent analysis 
due to their nominations being “non-materials” (i.e., not 
instructional materials). After this filtering, our analysis 
focused on 249 instructional materials nominated by 146 
afterschool sites. Of the 146 sites, 85 sites reported one 
material, 28 sites reported two materials, and 33 sites 
reported three or more materials. As shown in Table 1, a 
majority of these materials, 71.9%, were science 
enrichment materials and 28.1% of them were science 
curriculum materials. Of the 179 science enrichment 
materials reported, most were lessons/activities from either 
websites or books. Out-of-school and _ school-based 
curricula were nearly evenly divided and comprised most of 
the materials in the curriculum category. 

Use of materials by sites. We examined the range of 
materials that afterschool sites reported using and found 
that 106 (73%) of the 146 sites reported using at least one 
type of enrichment material while 61 sites (42%) reported 
using at least one type of curricular material. Regarding 
curriculum materials, 30 sites (20.5%) used out-of-school 
(Cl) materials, 28 sites (19.2%) used school-based 
materials (C2), and 6 sites (4.1%) used context-independent 
materials (C3). On the enrichment side, 47 sites (32.2%) 
used lessons/activities from websites (E1), 38 sites (26.0%) 
used lessons/activities from books (E2), and 29 sites 
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Table 1 
Categorization of Instructional Materials (N = 249) 
Category Code Material Type Frequency % 
Cl Out-of-school curricula 32 12.9 
Curriculum C2 School-based curricula 30 12.0 
C3 Context-independent 8 oe 
curricula 
El Lessons or activities 62 24.9 
from websites 
E2 Lessons or activities 50 20.1 
from books 
Enrichment E3 Trade book or media 7 2.8 
E4 Prepackaged science 21 8.4 
projects and kits 
ES Site-developed material 36 14.5 
or activity 
E General enrichment 3 2 


(source unknown) 





(20.0%) used their own site-developed materials (E5). The 
two other types of enrichment materials (E3 and E4) were 
used by a much smaller number of sites, 4.8% and 13.0%, 
respectively. 

Across the 146 sites that reported on material use in the 
survey, 40 sites (27.4%) used only curriculum materials, 
while 85 sites (58.2%) used only enrichment materials. Just 
21 sites (14.4%) used a combination of the two categories. 
In general, sites used only one category of material, even if 
they used multiple materials in their program, and this was 
typically multiple enrichment materials. There were 61 sites 
that reported using multiple materials at their site and about 
one third of these sites reported using both curricular and 
enrichment materials. 

Support Features of the Instructional Materials 

In our examination of the support features found in the 
instructional materials, we found that, on the whole, 
curricular materials had more support features than 
enrichment materials. Additionally, materials that had a 
relatively high number of support features from one 
dimension on the coding scheme usually had a relatively 
high number of support features in the other dimensions. In 
other words, most materials either had many of the support 
features of interest or very few support features. Of note is 
that there were some features that were rarely seen across 
the full spectrum of materials. Both curricular and 
enrichment materials, for instance, were found to have 
minimal support features addressing social-emotional 
learning goals (i.e., engagement, motivation, and attitude 
toward science); use of science notebooks or journals to 
record work; prompts to coordinate prior knowledge with 
new knowledge; supports or prompts to encourage 
reflection on learning; strategies for facilitators to encourage 
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children to listen and build on each others’ ideas; and 
opportunities for children themselves to design experiments 
or investigations. 

We performed a principle components analysis (PCA) on 
the dataset to see if we could reduce the dimensionality of 
the data. The first two dimensions only captured 32% of the 
variance of the dataset. Although the scores across the 
dimensions are correlated, which specific features materials 
are coded for are not—which is why the variance captured 
in the PCA is relatively low. 

We examined the main differences in support features 
across the categories and subcategories of instructional 
materials for each of the four dimensions of instructional 
support: (a) content and structure of the activities; (b) 
usability for facilitators and children; (c) engagement for 
children and relevance to their lives; and (d) promotion of 
scientific thinking, reasoning, and practice. The 
accompanying box and whisker plot for each of the four 
dimensions shows each of the subcategory’s median scores 
(i.e., the number of support features in that dimension) as 
well as the first and third quartiles and any outliers. 

Content and structure of the activities. Regarding the 
first dimension, content and structure of the activities, 
Figure 1 shows that lessons and activities in the two 
curriculum categories scored much higher on support 
features than the lessons and activities in the enrichment 
categories. This trend holds for all four of the dimensions, 
but it is especially pronounced for this dimension. Of 
particular note are the high scores among the school-based 
materials, which perhaps are to be expected on this 








Dimension 1: Content and Structure 


Enrichment 
From Books 


Enrichment 
From Websites 


Out-of-School 
Curriculum 


In-School 
Curriculum 


Figure 1, Box and Whisker plot for Dimension 1 (Content and Structure of 
the Activities) of the materials coding scheme, grouped by category. 
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dimension since it captures a focus on content and 
structure, which are hallmarks of materials for school 
settings. Among the lessons and activities from books (E2), 
Figure 1 shows a broad range on this dimension, 
highlighting the wide degree to which materials in this 
subcategory vary in terms of support features for content 
and activity structure. Overall, curriculum materials were 
more likely than enrichment materials to identify the age/ 
grade-level appropriateness of the lessons or activities, 
specify an expected time frame for completing them, and 
provide a clear sequence to the lessons or activities. Also of 
note is that curriculum materials were more likely to 
include scientific practices among learning goals. Other 
support features in this dimension were either similarly 
present or absent across the two categories of materials. 

Usability for facilitators and children. Regarding 
usability for facilitators and children, the second dimension, 
Figure 2 shows that the lessons and activities found in out-of- 
school and in-school curriculum materials scored much 
higher in support features than the lessons and activities from 
websites and books. Curriculum materials were much more 
likely to include notes and/or tips for facilitators within 
lessons or activities, activity sheets for children, and targeted 
support for English Language Learners. Curriculum materials 
were also more likely to suggest multiple ways (i.e., different 
modes) for children to engage in science, multiple grouping 
formats (e.g., pairs, whole group, individual) for children to 
work on tasks, and multiple ways for children to showcase 
their learning. The other support features were either not 
present in all of the materials or present in relatively similar 
frequencies in all of the materials. 


twennw mee 


Dimension 2: Usability for Facilities and Children 





Enrichment 
From Books 


Out-of-School 
Curriculum 


Enrichment 
From Websites 


In-School 
Curriculum 


Figure 2. Box and Whisker plot for Dimension 2 (Usability for Facilitators 
and Children) of the materials coding scheme, grouped by category. 
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Engagement for children and relevance to their 
lives. As illustrated in Figure 3, the lessons and activities in 
the two curriculum categories scored higher in support 
features for the third dimension, engagement for children 
and relevance to their lives, than the lessons and activities 
in the enrichment categories. Both enrichment categories 
were very low on this dimension, highlighting the lack of 
support for instructional options for children, and lack of 
guidance for facilitators to help engage children in science. 
Quite differently, curriculum materials were much more 
likely to situate the learning in an engaging context, such as 
a dramatic scenario, real-world situation, or compelling 
problem/issue. Lessons and activities in both types of 
curriculum materials were more likely to provide 
suggestions for facilitators on how to connect science to the 
real world, provide opportunities and support for children 
to make choices that are consequential for engagement and 
learning, and provide guidance for facilitators to help 
children make choices or carry out plans. Other support 
features in this dimension were similarly present or absent 
across the two categories of materials. 

Promotion of scientific thinking, reasoning, and 
practice. For the fourth dimension, promotion of scientific 
thinking, reasoning, and practice, Figure 4 shows that the 
lessons and activities across all the curriculum and 
enrichment categories scored much more similarly to each 
other than on the other three dimensions. Moreover, all four 
materials categories had high variation, especially out-of- 
school curriculum materials. Still, within this dimension, 
curriculum materials were more likely to provide guidance 


Dimension 3: Engagement for Children and Relevance 





Out-of-School 
Curriculum 


Enrichment 
From Websites 


In-School 
Curriculum 


Enrichment 
From Books 


Figure 3. Box and Whisker plot for Dimension 3 (Engagement for Children 
and Relevance to Their Lives) of the materials coding scheme, grouped by 
category. 


Volume 117 (6) 


Afterschool Science Instructional Materials 


12 


as 
oO 


@ 


Dimension 4: Promotion of Scientific Thinking 
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Figure 4. Box and Whisker plot for Dimension 4 (Promotion of Scientific 
Thinking, Reasoning and Practice) of the materials coding scheme, grouped 
by category. 


to facilitators to help children think, reason, and reflect 
about science; questions for facilitators to elicit ideas, 
questions, and contributions during discussion; and 
strategies for facilitators to encourage children to make 
claims, weigh evidence, collect, analyze, and interpret data, 
make and critique scientific explanations, and communicate 
or present findings. Other support features in this dimension 
were either similarly present or absent across the two 
categories of materials. 

Differences within categories. In our _ closer 
examination of the two types of materials from the 
enrichment category, several notable differences emerged. A 
rationale for learning goals was more likely to be included in 
lessons or activities from books (E2) than to be included in 
lessons or activities from websites (El). However, El 
materials were more likely to situate learning within a 
context and to include prompts to encourage or support 
learner interest in science than were E2 materials. 

There were also distinctions between the features likely 
to be included in out-of-school (C1) and school-based 
curricula (C2). C1 materials more often included materials 
for activities for the facilitators to use, materials for children 
to use, and prompts to encourage/support an interest in 
science. C2 materials more often included a rationale 
for learning goals, suggestions for adapting/extending 
activities, directions/procedures for children to follow, 
explanations situating learning in a context, opportunities 
for children to make choices consequential for learning, and 
guidance for facilitators to help children make choices or 
carry out plans. 
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Staff Use of Instructional Materials 

Our interviews with coordinators and facilitators across 
sites provided insight into how instructional materials were 
selected and used for planning and enacting science. Site 
coordinators and facilitators reported that they were guided 
by a number of considerations when reviewing possible 
lessons and activities. Considerations included the cost of 
resources needed to implement; whether the materials 
included activities that children would enjoy and be capable 
of doing; the alignment of the lessons or activities to the 
skills, knowledge, and interests of staff; and the complexity 
of planning and implementing with the materials. Site 
coordinators and facilitators emphasized that they reviewed 
materials with an eye toward selecting those that they 
believed would help staff enact engaging and meaningful 
science experiences with children. In this endeavor, they 
primarily focused attention on individual lessons and 
activities, and made judgments about how easy or difficult a 
“lift? each would be for staff to use to create a science 
experience. Site coordinators and facilitators, while being 
purposeful in selecting materials that they perceived would 
be supportive of staff, did not report that they looked for 
specific support features, such as pedagogical supports, 
within materials. Rather, they attended to broad aspects 
related to the feasibility of staff implementing the materials, 
such as clear instructions, familiar or easy-to-grasp science 
content, and minimal time for preparation and cleanup. 

The site coordinators and facilitators we interviewed all 
reported that they typically planned for short term, stand- 
alone science experiences. This was the case for 
coordinators and facilitators whose sites relied on 
enrichment materials as well as for those whose sites 
primarily used curricular materials. The general approach 
shared by coordinators and facilitators across sites was to 
identify lessons or activities within materials to support 
planning and implementation of one afterschool science 
session at a time. Some reported that they organized their 
selections around topics or themes, such as environmental 
sustainability, and sometimes chose multiple activities to 
match with a particular theme, but they did not select and 
plan activities for sequenced sessions that would explicitly 
link the learning experiences or build upon one another. 
Coordinators and facilitators who used primarily curricular 
materials reported that they used these materials flexibly and 
primarily as resources for finding stand-alone activities. 

Across sites, coordinators and facilitators reported that they 
conducted searches on the Internet to find lessons and 
activities. Those who mainly used enrichment materials at 
their sites identified the Internet as the primary source for 
lessons and activities, followed by science activity books. 
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Coordinators and facilitators who mostly relied on curricular 
materials also reported that they conducted Internet searches 
to find supplemental lessons and activities. Similarly, they 
typically did not search online for sequenced science 
experiences and instead sought to identify promising easy-to- 
implement stand-alone science activities. 


Discussion 

Afterschool site coordinators nominated a wide range of 
instructional materials that fell into two large categories: 
curricular materials and enrichment materials. They 
reported using primarily enrichment materials in the form 
of stand-alone lessons and activities selected from books 
and websites. In our analysis, we found that enrichment 
materials had fewer support features for facilitators and 
children that could aide in planning and enactment. 
Curricular materials, both those designed for school settings 
and those designed for out-of-school settings, were reported 
to be used much less frequently, yet were found in our 
analysis to be much more likely to have support features 
aligned with current recommendations for quality 
afterschool science (e.g., NRC, 2009) and with learning 
goals reported by site coordinators and facilitators. 

Of note is that not all of the enrichment materials we 
analyzed were lacking in support features. There were some 
materials that did include support features for facilitators 
and children. However, among those enrichment materials 
that did include support features we found that they varied 
greatly in the number and nature of the supports. By far, the 
predominance of enrichment materials used across 
afterschool sites included minimal support features. 

In our analysis of materials that were expressly designed 
for afterschool settings, we found that they often included 
support features that aimed to address the constraints and 
needs of the typical afterschool space. For instance, these 
materials tend to include flexible time frames for enacting 
activities, guidance for organizing and interacting with 
groups of children, science background information 
described for a layperson, and easy-to-learn strategies for 
encouraging children to collaborate and share ideas. One set 
of afterschool materials we reviewed included a collection of 
sequenced sessions so that children could build their science 
understanding over time, but these same materials were also 
designed so that children’s participation in any one session 
was not dependent upon completing the prior session. In this 
way, the materials accommodated flexible attendance—an 
important characteristic of the afterschool setting. 

Despite having an attractive range of support features, we 
found that materials expressly designed for afterschool 
settings were not widely used across afterschool sites. From 
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our interview data with site coordinators and our past case 
studies of afterschool programs (Lundh, House, Means, & 
Harris, 2013), we know that there are many factors that site 
coordinators weigh when deciding on instructional materials. 
These factors include cost and ease of access to the 
materials, perceived time and resources needed to plan and 
implement, alignment to the skills and interests of 
facilitators, and the enjoyment level for children. Many of 
the afterschool materials in our review were found to include 
features that appeared to match well with what site 
coordinators reported as important for the high-quality 
implementation of science at their sites. Yet, we also know 
that site coordinators and facilitators tended to plan for the 
short term and preferred stand-alone science activities. They 
conducted searches on the Internet to identify and develop or 
refine activity ideas rather than to identify sequenced 
materials or comprehensive instructional resources. Thus, 
the low uptake of afterschool instructional materials may be 
due in part to limited awareness of the value of them. Our 
research shows that by relying primarily on stand-alone 
activities from books or websites, facilitators likely miss out 
on support features that could help them enact meaningful 
science experiences with children. 

It is also important to emphasize that while a particular 
instructional material might score high across multiple 
dimensions it does not necessarily follow that the material 
will be easily enacted by afterschool staff. The usability of 
instructional materials in real-world afterschool settings may 
depend on a range of factors, such as facilitator preparedness, 
time for planning and enactment, and fit with program goals 
and routines. For these reasons, it would not be suitable to 
select materials solely on the basis of dimension scores. 
Rather, we hold that the dimensions can provide good 
guidance when reviewing materials for their fit with current 
perspectives on how learners can best be supported in out-of- 
school settings, as described in the NRC’s synthesis report 
(NRC, 2009) on informal science learning. 

An important limitation of our study is that we did not 
examine how the materials were actually implemented at 
afterschool sites, nor did we examine whether or not 
facilitators were using the support features that were available 
to them and/or what features were most beneficial and in 
what ways. In recent years, the science education community 
has been building a rich knowledge base on the role of 
educative features in curriculum materials (e.g., Arias, 
Bismack, Davis, & Palincsar, 2016; Wilson, Schweingruber, 
& Nielsen, 2015). Much of this work has focused on school 
science curriculum materials. While there is much that can be 
learned from formal settings, there is a great need for studies 
on instructional materials and their effectiveness in informal 
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environments. Such research could help the science 
education community better understand the ways in which 
materials can be designed to effectively support science 
learning experiences in everyday afterschool programs. 


Conclusions 

There is broad agreement that afterschool has 
tremendous potential for offering rich science learning 
experiences for children. Well-designed instructional 
materials, with features that align with current perspectives 
of what makes for quality afterschool science, could help 
afterschool facilitators and children realize that potential. Of 
course, instructional materials are but one component of a 
larger system of resources necessary for ensuring high- 
quality science programming. They are, however, an 
essential part of that system and if designed with the unique 
affordances of the afterschool space in mind, they hold 
promise for supporting facilitators in implementing 
meaningful experiences that can help all children become 
engaged and enthusiastic learners of science. 
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Appendix: Materials Coding Scheme 
Dimension 1. Content and Structure of Activities 





Feature 


Indicator 





1.1. Science learning goals: 
Does the material include explicit 
learning goals? 


1.2 Activities: Does the material 
include one or more activities and 
are they sequenced to help children 
make progress toward achieving 
key science learning goals? 


1.3 Time for doing science: Does 
the material allocate time for 
learning science via investigations/ 
projects? 


Code 
PAN 
ih gal NV, 
eAS! 
1.1.B1. 
(leB2: 
in iBs: 
Ree 
1D: 
28 
1.2.B. 
LAG 
lea: 
1:3.A; 
3h 18), 


Identifies science content among its learning goals 

Identifies science practice among its learning goals 

Identifies engagement, motivation, attitude toward science among its learning 

oals 

sctnck content learning goals are explicitly aligned to benchmarks/standards 

Science practice learning goals are explicitly aligned to benchmarks/standards 

Engagement, motivation, attitude toward science learning goals are explicitly 
aligned to benchmarks/standards 

Provides a rationale for the learning goal(s) 

The age/grade level is specified 

Stand-alone activity 

The activity is part of an organized group of activities (note how grouped, e.g., 
by topic, theme, chapter etc.) 

The activity is part of a logical sequence of activities that build toward more 
sophisticated understanding of science content and/or scientific practices 

Provides a rationale for the sequence of activities 

The activity extends beyond one session or connects to a prior or next activity 

Time frame is specified in the material for carrying out activities/tasks 


Dimension 2. Usability for Facilitators and Children 


Feature 


2.1 Guidance for facilitators: Does 
the material include guidelines for 
enactment by the facilitator? 


2.2 Educative features: Does the 
material include background 
information to support facilitator 
knowledge and practice? 

2.3 Support for children: Does the 
material include supports for 
children to engage in scientific 
activity? 


2.4 Accessibility for diverse 
populations: Does the material 
provide multiple means for diverse 
populations to access the science? 
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Indicator 


Overview of the activity is included 

Describes materials and necessary preparation 

Provides materials for activities 

Step-by-step guidance for how to enact 

Safety guidelines 

Prompts for discussion/reflection 

Suggestions for adapting or extending activities 

Background on content knowledge (e.g., glossary) 

Background on children’s likely ideas, including misconceptions 

Notes/tips for facilitator (e.g., notes/tips for effective instruction, management, 
etc.) 

Provides science materials for children to use during activities 

Activity provides firsthand experiences with phenomena 

Activity provides reading about science 

Activity provides secondhand experiences (e.g., demonstration) 

Directions/Procedures for children to follow themselves 

Activity sheets are included 

Science notebook or journal for children to record work is included 

More than one way for children to learn (e.g., print materials, hands-on, video, 
etc.) 

More than one way for children to work (e.g., individual/collaborative) 

More than one way for children to showcase their learning 

Supplements or guidance for facilitators to adapt activities for children with 
diverse needs 

Supplements or guidance for facilitators to meet needs of English language 
learners (ELLs) 

Support for ELL in the materials provided to children (e.g., illustrations, 
translations, etc.) 
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Dimension 3. Engagement for Children and Relevance to Their Lives 





Feature 


Code 


Indicator 





3.1 Connections to children’s 
everyday lives: 

Does the material attempt to bridge 
science to children’s everyday 
social and physical world? 


3.2 Leverage children’s prior 
knowledge and interests: Does the 
material provide prompts and 
support for facilitators to help 
children use their prior knowledge 
and interests for learning? 

3.3 Choice for children: Does the 
material include opportunities and 
support for children (individual or 
group) to make choices about the 
direction of their learning? 


ep el 
3.1.3. 


Bl 
ool: 


Includes science tasks or activities that relate to children’s everyday lives 

Situates learning in a context, such as a driving question, dramatic scenario, real- 
world situation, or compelling problem/issue 

Provides suggestions for community involvement and/or connections to life at 
home 

Provides suggestions for facilitators about how to connect science to the real 
world 

Provides elicitation prompts for children’s prior knowledge and interests 

Prompts for children to relate their prior knowledge and own ideas/interests to 
the activities 

Prompts to encourage children to coordinate their prior knowledge with new 
knowledge 

Prompts to encourage/support children’s developing interests in science 

Includes opportunities for children to make choices that are consequential for 
engagement 

Includes opportunities for children to make choices that are consequential for 
learning 

Includes guidance for facilitators to help children make choices and/or carry out 
their decisions/plans 


Dimension 4. Promotion of Scientific Thinking, Reasoning, and Practice 


Feature 


4.1 Thinking, reasoning, and 
reflection: Does the material 
provide supports/prompts for 
children to think, reason, and reflect 
about science experiences? 

4.2 Collaborative discourse: 

Does the material provide questions 
or strategies for facilitators to 
encourage productive conversation? 


4.3. Scientific inquiry practices: 
Does the material include activities 
that engage children in a rdnge of 
scientific inquiry practices? 


Indicator 


Supports/prompts for children to question, predict, observe, and/or explain 

Supports/prompts for children to reflect on their learning experiences 

Guidance for facilitators to help children to think, reason, and reflect about 
science experiences 


Questions for facilitators to use to elicit participant ideas, questions, and 
contributions during discussion 

Strategies for facilitators to encourage children to listen to and build upon one 
another’s ideas 

Strategies for facilitators to encourage children to make claims and weigh 
evidence during discussion 

Pose and/or pursue researchable or investigable questions 

Design experiments and/or investigations 

Conduct experiments and/or investigations 

Collect, analyze, and interpret data 

Build and/or use models to represent phenomena, test designs, or show ideas/ 
understandings 

Make and critique scientific explanations 

Communicate/present findings 
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PROBLEMS 


sec ee i n Ted Eisenberg, Section Editor 


This section of the Journal offers readers an opportunity to exchange interesting mathematical problems and 
solutions. Please send them to Ted Eisenberg, Department of Mathematics, Ben-Gurion University, Beer-Sheva, Israel 
or fax to: 972-86-477-648. Questions concerning proposals and/or solutions can be sent e-mail to eisenbt@013.net. 
Solutions to previously stated problems can be seen at http://www.ssma.org/publications. 


Solutions to the problems stated in this issue should be posted before December 15, 2017 


¢ 5457: Proposed by Kenneth Korbin, New York, NY 
Given angle 4 with sin A= i. A circle with radius 1 and a circle with radius x are each tangent to both sides of the 
angle. The circles are also tangent to each other. Find x. 


¢ 5458: Proposed by Michal Kremzer, Gliwice, Silesia, Poland 
Find two pairs of integers (a, b) from the set {1,2,3,4,5,6, 7,8, 9} such that for all positive integers n, the number 


c=537aaab...b 18403 
ae 


is composite, where there are 2n numbers b between a and 1 in the string above. 


* 5459: Proposed by Arsalan Wares, Valdosta State University, Valdosta, GA 

Triangle ABC is an arbitrary acute triangle. Points X, Y, and Z are midpoints of three sides of AABC. Line segments 
XD and XE are perpendiculars drawn from point X to two of the sides of AABC. Line segments YF and YG are perpen- 
diculars drawn from point Y to two of the sides of AABC. Line segments Z/J and ZH are perpendiculars drawn from 
point Z to two of the sides of AABC. Moreover, P=ZJ 1 FY, Q=ZH M DX, and R=YG ™XE. Three of the triangles 
and three of the quadrilaterals in the figure are shaded. If the sum of the areas of the three shaded triangles is 5, find the 
sum of the areas of the three shaded quadrilaterals. 





282 Volume 117 (6) 


Problems 


* 5460: Proposed by Angel Plaza, Universidad de Las Palmas de Gran Canaria, Spain 
Ifa,b > 0 and x,y > 0 then prove that 


a Fs b a’ +b? 
ax Dy. wONTOY © ax y 











* 5461: Proposed by José Luis Diaz-Barrero, Barcelona Tech, Barcelona, Spain 
Compute the following sum: 


~. cos (2n—1) 
Di (2n-1)? — 


n=1 


* 5462: Proposed by Ovidiu Furdui, Technical University of Cluj-Napoca, Cluj-Napeca, Romania 
Let n > 1 be an integer. Calculate 


I cos x oe 
0 (1+,/sin (2x))" 
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